One line description of the issue

Problems using Django 3.x in initial migration

When using Django major version 3 I get migration errors (makmigration works but migrate fails) on a fresh mongo db installation (docker-compose clearing volume data):

Operations to perform:
  Apply all migrations: admin, auth, authtoken, contenttypes, core, sessions
Running migrations:
[DEBUG-DJANGO] 2021-01-14 15:13:58,219 django.db.backends.schema  schema in              execute() line 133: CREATE TABLE "core_user" ("id" int NOT NULL PRIMARY KEY AUTOINCREMENT, "password" string NOT NULL, "last_login" date NULL, "is_superuser" bool NOT NULL, "keycloak_id" string NOT NULL UNIQUE, "email" string NOT NULL UNIQUE, "is_active" bool NOT NULL, "is_staff" bool NOT NULL); (params None)
[DEBUG-DJANGO] 2021-01-14 15:13:58,223 django.db.backends  utils in            debug_sql() line 130: (0.003) QUERY = 'CREATE TABLE "core_user" ("id" int NOT NULL PRIMARY KEY AUTOINCREMENT, "password" string NOT NULL, "last_login" date NULL, "is_superuser" bool NOT NULL, "keycloak_id" string NOT NULL UNIQUE, "email" string NOT NULL UNIQUE, "is_active" bool NOT NULL, "is_staff" bool NOT NULL)' - PARAMS = (); args=None
  Applying core.0001_initial...Traceback (most recent call last):
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 856, in parse
    return handler(self, statement)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 897, in _create
    query = CreateQuery(self.db, self.connection_properties, sm, self._params)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 645, in __init__
    super().__init__(*args)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 84, in __init__
    super().__init__(*args)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 62, in __init__
    self.parse()
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 729, in parse
    self._create_table(statement)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 657, in _create_table
    self.db.create_collection(table)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/pymongo/database.py", line 365, in create_collection
    raise CollectionInvalid("collection %s already exists" % name)
pymongo.errors.CollectionInvalid: collection core_user already exists

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/cursor.py", line 56, in execute
    params)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 783, in __init__
    self._query = self.parse()
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/sql2mongo/query.py", line 884, in parse
    raise exe from e
djongo.exceptions.SQLDecodeError:

	Keyword: None
	Sub SQL: None
	FAILED SQL: CREATE TABLE "core_user" ("id" int NOT NULL PRIMARY KEY AUTOINCREMENT, "password" string NOT NULL, "last_login" date NULL, "is_superuser" bool NOT NULL, "keycloak_id" string NOT NULL UNIQUE, "email" string NOT NULL UNIQUE, "is_active" bool NOT NULL, "is_staff" bool NOT NULL)
	Params: None
	Version: 1.3.3

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql)
  File "/home/christian/.venvs/backend-ZIt4lE0c/lib/python3.7/site-packages/djongo/cursor.py", line 59, in execute
    raise db_exe from e
djongo.database.DatabaseError

The above exception was the direct cause of the following exception:

Once the initial migrations are successfully done, I can upgrade to a django 3 installation with a pipenv that looks like that and everything works fine:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
django = "==3.0.5"
django-cors-headers = "==3.4.0"
djangorestframework = "*"
drf-yasg = {extras = ["validation"], version = "*"}
djongo = "*"
jsonfield = "*"
python-keycloak = "*"
numpy = "*"
qrcode = "*"
scipy = "*"
audiofile = "*"
python-magic = "*"
pymongo = "==3.7.2"
names = "*"
uwsgi = "*"
flake8 = "*"
pillow = "*"
requests-toolbelt = "*"
sqlparse = "==0.2.4"

[requires]
python_version = "3.7"

My question: How to set up a django virtual environment that uses Django 3.x for initial migration?

That's a good request. There has already been a similar request for simple dict field as well. Will tag this as a new feature request that can be used to extend the overall feature capabilities. Thanks … On 09-Jan-2018 1:27 PM, "Samuel Bishop" ***@***.***> wrote: Simple flat arrays of values stored in an Array value are currently unsupported. Ideally, in addition to the existing support for Document Arrays, it would be good to have 'simple' array support. e.g.: Support for https://docs.mongodb.com/manual/tutorial/query-arrays/ in addition to the current support for https://docs.mongodb.com/manual/tutorial/query-array- of-documents/ I can work around this in most circumstances by using the manager query method and properties, but its a hinderance to interoperability with existing MongoDB databases, and could arguably be better. Especially in light of the built in support Django has for PostgreSQL array fields. https://docs.djangoproject.com/en/2.0/ref/contrib/ postgres/fields/#arrayfield — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#44>, or mute the thread https://github.com/notifications/unsubscribe-auth/AQO4BChrPXgrNXfYocv-WY63AbDLBJcuks5tIxvbgaJpZM4RXcPQ .

why this still in TODO?

fixed in this PR #474

please merge this PR and release it. in my case, I need array built-in.

I need help! Same issue here

Error upon attempting to python3 manage.py runserver

Watching for file changes with StatReloader
Performing system checks...

Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'djongo'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 117, in inner_run
    self.check(display_num_errors=True)
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/core/management/base.py", line 395, in check
    include_deployment_checks=include_deployment_checks,
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/core/management/base.py", line 382, in _run_checks
    return checks.run_checks(**kwargs)
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/home/me/Desktop/testdjongoerror/ztest/lib/python3.6/site-packages/django/core/checks/model_checks.py", line 34, in check_all_models
    errors.extend(model.check(**kwargs))
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/db/models/base.py", line 1255, in check
    *cls._check_long_column_names(),
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/db/models/base.py", line 1774, in _check_long_column_names
    connection = connections[db]
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/db/utils.py", line 207, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/me/Desktop/retestdjongoerror/ztest/lib/python3.6/site-packages/django/db/utils.py", line 126, in load_backend
    ) from e_user
django.core.exceptions.ImproperlyConfigured: 'djongo' isn't an available database backend.
Try using 'django.db.backends.XXX', where XXX is one of:
    'mysql', 'oracle', 'postgresql', 'sqlite3'

This is the DB settings from when the errors started. myapp/settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    },
    'analytics': {
          'ENGINE': 'djongo',
          'NAME': 'demodb1',
          'CLIENT': {
                'host': 'mongodb://mongodb:27017',
                'username': 'user1',
                'password': 'pass1',
                'authSource': 'admin1',
                'authMechanism': 'SCRAM-SHA-1',
                }
        },
}

pip freeze:

asgiref==3.3.1
dataclasses==0.8
Django==2.2.8
djongo==1.3.3
pymongo==3.11.2
pytz==2020.5
sqlparse==0.2.4

can you show any logs of your errors

Would you mind sharing your docker setup? I am attempting to Dockerize a small django app with mongodb/djongo and running into issues...

On Tue, Jan 5, 2021 at 7:10 AM Leo Lafon notifications@github.com wrote:

The same issue arose recently for me when building a docker image inside a deployment pipeline. We fixed it by switching from:

FROM python:3.6

to

FROM python:3.6.8

in the Dockerfile.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nesdis/djongo/issues/171#issuecomment-754695029, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPP5AUDMSAPGD3EJFNRMODSYMTURANCNFSM4FP2623A .

I have the same issue. I'm using virtualenv on ubuntu. After lots of search, this was my last bus stop: upgrading django==2.2.8 --> django==3.1.5 I have these:

Django==3.1.5
djongo==1.3.3
Python 3.6.9

Worked!

The same issue arose recently for me when building a docker image inside a deployment pipeline. We fixed it by switching from:

FROM python:3.6

to

FROM python:3.6.8

in the Dockerfile.

Model.objects.all() returns only one record

Using django 2.0.6, pymongo 3.7.2, djongo 1.2.29, connects to database correctly, and returns all rows when using the model below.

class EventModel(models.Model): _id = models.ObjectIdField() event_id = models.IntegerField(primary_key=True)

using django 3.0.5, pymongo 3.11.2 and djongo 1.3.3, the model

1. gives an error on multiple primary keys and
2. when removing the primary_key=True, returns only one record on EventModel.objects.all()

I am using Django rest framework social oauth2 for authentication in one Django app. For development purposes, I have used this package with Sqlite3 (SQL database) and it works perfectly. However, now I want to migrate this Django app to MongoDB (NoSQL database) via using Djongo. After implementing all the required changes it is showing this bizarre error. I have seen everywhere couldn't find a relevant solution

After adding this command:

**python manage.py migrate**

Operations to perform:
Apply all migrations: admin, auth, contenttypes, oauth2_provider, sessions, social_django

Running migrations:

Not implemented alter command for SQL ALTER TABLE "oauth2_provider_accesstoken" ADD COLUMN "source_refresh_token_id" long NULL UNIQUE

Applying oauth2_provider.0001_initial...Traceback (most recent call last):
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/cursor.py", line 51, in execute
self.result = Query(
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 783, in init
self._query = self.parse()
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 875, in parse
raise e
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 856, in parse
return handler(self, statement)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 888, in _alter
query = AlterQuery(self.db, self.connection_properties, sm, self._params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 425, in init
super().init(*args)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 84, in init
super().init(*args)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 62, in init
self.parse()
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 435, in parse
self._add(statement)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/sql2mongo/query.py", line 598, in _add
raise SQLDecodeError(err_key=tok.value,
djongo.exceptions.SQLDecodeError:

Keyword: long
Sub SQL: ALTER TABLE "oauth2_provider_accesstoken" ADD COLUMN "source_refresh_token_id" long NULL UNIQUE
FAILED SQL: ('ALTER TABLE "oauth2_provider_accesstoken" ADD COLUMN "source_refresh_token_id" long NULL UNIQUE',)
Params: ([],)
Version: 1.3.3

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/cursor.py", line 59, in execute
raise db_exe from e
djongo.database.DatabaseError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "manage.py", line 22, in
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/core/management/init.py", line 401, in execute_from_command_line
utility.execute()
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/core/management/init.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/core/management/base.py", line 328, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/core/management/base.py", line 369, in execute
output = self.handle(*args, **options)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 231, in handle
post_migrate_state = executor.migrate(
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/migrations/executor.py", line 245, in apply_migration
state = migration.apply(state, schema_editor)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 110, in database_forwards
schema_editor.add_field(
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 480, in add_field
self.execute(sql, params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 142, in execute
cursor.execute(sql, params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/utils.py", line 90, in exit
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
return self.cursor.execute(sql, params)
File "/home/anakin/virtual_envs/js_backend/lib/python3.8/site-packages/djongo/cursor.py", line 59, in execute
raise db_exe from e
django.db.utils.DatabaseError

Thanks for your consideration. Really appreciate it.

the type int and long was not handled

@khikmatullaev Unfortunately, no.

@NOBLEGG I have faced the same problem when I wanted to use oauth2_provider with mongodb (djongo). Could you find a solution?

Hi guys i had same error in

 django==3.0.7    
 djongo==1.3.2

then i uninstall bson & pymongo and reinstall pymongo and finally fixed!

 pip uninstall bson
 pip uninstall pymongo
 pip install -U pymongo 

try to change the version of djongo 1.3.2 to 1.3.3, this solved the problem in my case.

Hello how are you, someone knows if you can use the Djongo library to connect to the AWS DynamoDB database.

I know that the Pynamodb library can be used but it would not be connected to the django ORM

One line description of the issue

When I store any object in my database and retrieve it again afterwards it's always returned as (None). However, when I check out the actual database using MonoDB Compass I can see all objects the way I saved them.

My Script: OS = MacOS 11.0.1 Python==3.8.2 Django==3.0.5 djongo==1.3.3 pymongo==3.11.2 MongoDB 4.4 Community Edition

# database setting
DATABASES = {
    'default': {
        'ENGINE': 'djongo',
        'NAME': 'martini',
        'ENFORCE_SCHEMA': False,
        'LOGGING': {
            'version': 1,
            'loggers': {
                'djongo': {
                    'level': 'DEBUG',
                    'propagate': False,                        
                }
            },    
        },
    }
}

# my model
class Transaction(models.Model):
    transaction_type = models.CharField(max_length=10,default="")
    traded_player_id = models.CharField(max_length=10,default="")
    traded_player_name = models.CharField(max_length=30,default="")
    value = models.IntegerField(default=-1)

# storing the model
transaction = Transaction(
    transaction_type="SALE",
    traded_player_id="2839",
    traded_player_name="Grill",
    value=499700
)
transaction.save()

Issue

Checking out the shell in-code or using django's shell I receive the following result:

>>>Transaction.objects.filter()
<QuerySet [<Transaction: Transaction object (None)>, <Transaction: Transaction object (None)>, <Transaction: Transaction object (None)>]>

However when I take a look at the database in MongoDB Compass:

I have exactly the same problem

I have the same issue. If directories is a list, I can't assign a model instance :-(

@brunowego Thanks alot, but in my case it helps with djongo 1.3.1, but in djongo 1.3.3 it still drops the same error

This error occurs because the django utils.py file does not recognize that django is an available backend, to solve this error , please follow these steps... 1 . Go to C:\Users\User\AppData\Local\Programs\Python\Python38-32\Lib\site-packages and find the djongo folder . 2. Now cut and paste the djongo folder in the C:\Users\aman\AppData\Local\Programs\Python\Python38-32\Lib\site - packages\django\db\backends directory . 3. Now u might be getting the error cannot import six from django.utils. For resolving that go to C:\Users\aman\AppData\Local\Programs\Python\Python38-32\Lib\site - packages\django\db\backends\djongo\operations.py file and replace the line from django.utils import six , datetime to from django.utils import datetime and beneath that just write import six . 4. Now in the settings.py file of your django project add the lines DATABASES = { 'default': { 'ENGINE': 'django.db.backends.djongo', 'NAME': 'your-database-name', } } 5. That should resolve the problem ....

Originally posted by @aaman123 in https://github.com/nesdis/djongo/issues/171#issuecomment-571087529

It is pretty easy to just shout a lot of insults and curses, But that picture tells nothing.

I just added ID to one admin model and worked out of the box, so you might be doing something wrong, but nobody will help you with such attitude and without giving more information about how you set up everything.

Working on it

On Tue, Dec 8, 2020 at 6:35 AM mostafa khaki notifications@github.com wrote:

any activity? @johndoe434 https://github.com/johndoe434 @nesdis https://github.com/nesdis

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nesdis/djongo/pull/489#issuecomment-740566434, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQHWNEXJEX7UQKDZPTDPQ5DSTYFO7ANCNFSM4TDWQV4Q .