profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/jhs/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Jason Smith jhs McKinsey & Company New York, NY CouchDB committer; Software Architect, business builder at McKinsey & Company

apache/nano 1126

Nano is now part of Apache CouchDB. Repo moved to https://GitHub.com/apache/couchdb-nano

jhs/cqs 124

CouchDB Queue Service: API-compatible Amazon SQS implementation with NodeJS, CouchDB

jhs/build-couchdb 65

Batteries-included CouchDB build system

jcrugzz/tcp-proxy 43

A simple tcp proxy with pluggable duplex streams for custom protocol proxying

jhs/app-engine-console 25

A Python console in Google's App Engine. Great for debugging and testing

jhs/bigdecimal.js 24

Javascript BigDecimal data type for arbitrary precision real numbers

jhs/audit_couchdb 17

Detect security issues in an Apache CouchDB server

jhs/couchdb 15

Mirror of Apache CouchDB

jhs/davcouch 11

CouchDB as a WebDAV server, all from a CouchApp

issue closedapache/couchdb-nano

Support over 1k changes feed listeners

<!--- Provide a general summary of the issue in the Title above --> Hello. I have a question regarding Nano performance: Is it possible to use Reading Changes Feed (aka nano.changesReader) to track the changes in parallel for a large number (1k...10k) of databases in a single CouchDB instance? Currently I see the big performance slowdown, as it takes minutes (10-20 min) to pick-up the latest change in any db. Thanks.

Expected Behavior

<!--- If you're describing a bug, tell us what should happen --> <!--- If you're suggesting a change/improvement, tell us how it should work --> No performance issues in case of 1k parallel changes feed listeners.

Current Behavior

<!--- If describing a bug, tell us what happens instead of the expected behavior --> <!--- If suggesting a change/improvement, explain the difference from current behavior --> It takes minutes to pick-up the latest change in any db

Steps to Reproduce (for bugs)

<!--- Provide a link to a live example, or an unambiguous set of steps to --> <!--- reproduce this bug. Include code to reproduce, if relevant -->

  1. create 1k dbs
  2. create 1k EventEmitter's via
nano.use(dbName).changesReader
                .start({ includeDocs: true })
                .on("change", changes => {
                    console.log(`change in ${dbName}: `, changes);
                })
                .on('error', (error) => {
                    console.log(` error in ${dbName}`, error);
                });

Context

<!--- How has this issue affected you? What are you trying to accomplish? --> <!--- Providing context helps us come up with a solution that is most useful in the real world --> Related use-case - messaging app. Each user has a message db, with outgoing and incoming messages. I want to track the changes when I send a message to somebody (here the message stored in db via PouchDB - CouchDB sync process), lookup the destination and write this message in destination user db.

Your Environment

<!--- Include as many relevant details about the environment you experienced the bug in -->

  • Version used: couchdb:3.1; "nano": "^9.0.3"
  • Operating System and version (desktop or mobile): ubuntu 18.04

closed time in 15 hours

BrodaUa

issue commentapache/couchdb-nano

Support over 1k changes feed listeners

There's a number of things going on here:

  • connection pooling in Node.js - Nano defaults to 50 concurrent connections between a client and a server on the same domain name (see https://github.com/apache/couchdb-nano/blob/main/lib/nano.js#L25). So even if you have 1000 things (in this case changes feed listeners) running, there would only be 50 in flight at any one time. Upping this value could give you a faster response.
  • how many changes feed can CouchDB handle. That rather depends how big a CouchDB cluster you have, the hardware it's running on, the rate of change of data, the size of the documents and it will definitely by dependent on the number of concurrent changes feeds in flight at any one time.

In short, if you're relying on this mechanism to scale up your application (e.g. one changes-feed/database per user) then eventually you will run out of CouchDB capacity. What that number is (100, 1k, 10k, 100k?) is difficult to estimate theoretically but you should be able to run experiments to see when performance tails off. If you have a fixed number of users (e.g. employees in shop) then you might be ok - if you expect numbers to increase over time (like a social network), then you might have to go back to the drawing board.

BrodaUa

comment created time in 15 hours

issue openedapache/couchdb-nano

Support over 1k changes feed listeners

<!--- Provide a general summary of the issue in the Title above --> Hello. I have a question regarding Nano performance: Is it possible to use Reading Changes Feed (aka nano.changesReader) to track the changes in parallel for a large number (1k...10k) of databases in a single CouchDB instance? Currently I see the big performance slowdown, as it takes minutes (10-20 min) to pick-up the latest change in any db. Thanks.

Expected Behavior

<!--- If you're describing a bug, tell us what should happen --> <!--- If you're suggesting a change/improvement, tell us how it should work --> No performance issues in case of 1k parallel changes feed listeners.

Current Behavior

<!--- If describing a bug, tell us what happens instead of the expected behavior --> <!--- If suggesting a change/improvement, explain the difference from current behavior --> It takes minutes to pick-up the latest change in any db

Steps to Reproduce (for bugs)

<!--- Provide a link to a live example, or an unambiguous set of steps to --> <!--- reproduce this bug. Include code to reproduce, if relevant -->

  1. create 1k dbs
  2. create 1k EventEmitter's via
nano.use(dbName).changesReader
                .start({ includeDocs: true })
                .on("change", changes => {
                    console.log(`change in ${dbName}: `, changes);
                })
                .on('error', (error) => {
                    console.log(` error in ${dbName}`, error);
                });

Context

<!--- How has this issue affected you? What are you trying to accomplish? --> <!--- Providing context helps us come up with a solution that is most useful in the real world --> Related use-case - messaging app. Each user has a message db, with outgoing and incoming messages. I want to track the changes when I send a message to somebody (here the message stored in db via PouchDB - CouchDB sync process), lookup the destination and write this message in destination user db.

Your Environment

<!--- Include as many relevant details about the environment you experienced the bug in -->

  • Version used: couchdb:3.1; "nano": "^9.0.3"
  • Operating System and version (desktop or mobile): ubuntu 18.04

created time in 16 hours

issue commentpouchdb/pouchdb

Plan for next major release?

For all folks interested in collaborate and talk about Pouchdb. I invite you to join the channel at the Official Couchdb workspace: https://couchdb.slack.com/archives/C016TJAE7A4

shadow81627

comment created time in 3 days

issue commentpouchdb/pouchdb

Plan for next major release?

No problem @daleharvey, let me know if I can be of any assistance.

shadow81627

comment created time in 3 days

issue commentpouchdb/pouchdb

Plan for next major release?

Cheers for the ping, promise this is getting cleaned up soon :)

shadow81627

comment created time in 4 days

IssuesEvent

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Hey @deepmouse Thank you very much for taking the time to document this. All of us are learning and obviously, can't say we know everything. I do appreciate your input. Pouchdb is planning a major release and hope that this kind of ideas could improve the technology. I have to be very honest with you, this would not be easy or fast. We must have a realistic expectation, maybe this will be scrapped, but taken into account. Cheers.

deepmouse

comment created time in 4 days

issue commentpouchdb/pouchdb

Plan for next major release?

Hey @daleharvey I wonder if you can consider open this issue again.

Thanks.

shadow81627

comment created time in 4 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Also, I am not quite sure what was meant by "having to be opinionated". The P2P protocol would be only responsible for establishing the connection between two browsers. Whatever goes through that connection would be up to PouchDB

deepmouse

comment created time in 4 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

P2P protocol ecosystem being chaotic, I am not sure what that means. There's plenty of choices out there that are stable, and even if they move fast, the support can be pinned to a stable version that is known to work.

Regarding bundle size, some of these implementations are only a few kB, which would not be much compared to the bundle size of PouchDB itself. Alternatively, PouchDB could include its own adapter that talks to the P2P network without any external code introduced as a dependency. Of course, if the P2P adapter is not used, it's 0 bytes.

Regarding reliability of P2P networks:

  • You can run your own P2P node if you want. You would be in as much control of it as a regular database server
  • You don't have to choose between P2P and central server. You can mix and match, use central server and fall back to P2P if it goes down or vice versa.
  • If you cannot run your own P2P node or your own central server, right now you cannot share data between devices at all. Even in the worst case scenario of P2P connection not being reliable, a non-reliable way of sharing data is preferable to not being able to share data at all.

Many people have implemented various adapters for different databases that demonstrate how P2P sync works. The problem is, as long as they are not officially supported, they tend to break or the author moves on to other projects, leaving it abandoned. Very different from a first-party support that "just works" because it is supported.

Obviously it's the choice of the main contributors to PouchDB to decide whether to include a browser<->browser sync strategy and if so, how to do it. The value for app authors and users is clearly there, and it seems that all the building blocks are ready. It's just a matter of putting them together. Might not be PouchDB today, but I bet this functionality will be soon in many databases as offline-first becomes bigger, people are getting skeptical about hosting everything on big cloud providers and are worried about data privacy.

deepmouse

comment created time in 4 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Great insights as always Diana @garbados , your feedback is greatly appreciated. What do you think @deepmouse ?

deepmouse

comment created time in 5 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

In my experience, mapping objects like distributed log entries to PouchDB documents requires asserting some opinions. This makes it hard to say that there is one correct way to do that mapping, which makes this problem perfect for plugin developers to handle as they have the freedom to form and apply those opinions.

deepmouse

comment created time in 5 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

@RangerMauve wrote pouchdb-adapter-hyperbee, a great example P2P adapter plugin. It allows you to use hyper:// addresses, though hypercore archives can only have one writer which means that if you aren't the author of a hypercore archive, write methods like .put() will fail. (It is possible to instrument multi-writer hypercores but the tooling for it is a bit limited at the moment.)

If you'd like to try writing some P2P protocol adapters, I'd recommend starting with that example :)

deepmouse

comment created time in 5 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Practically speaking, we're talking about adding new adapters. Imagine this code snippet that replicates between a hypercore and a CouchDB installation:

PouchDB.replicate('hypercore://[key]', 'http://[user]:[pass]@localhost:5984/my-db')

Internally, PouchDB would try and use a hypercore adapter to read the entries stored in its log, and then send them to the indicated CouchDB installation. If you add a hypercore adapter, PouchDB will automatically associate it with hypercore:// addresses.

P2P technologies often have specific environmental requirements, like access to networking capabilities that aren't available in browsers without a lot of tooling, or require bundling software that could have a significant impact on the size of PouchDB in its transpiled form. For this reason I'd recommend writing protocol-specific adapter plugins that you might use like this:

PouchDB.plugin(require('your-hypercore-plugin'))

Then you can include these adapters in your application as needed.

I would not recommend bundling such adapters into PouchDB itself at this time, as the P2P protocol ecosystem is... very chaotic.

deepmouse

comment created time in 5 days

pull request commentpouchdb/pouchdb

feat: support partial_filter_selector

OK, now I get different failures, which I am able to reproduce locally. Further commits coming soon...

garbados

comment created time in 5 days

issue commentpouchdb/pouchdb

Implement partial_filter_selector?

Thank you very much @daleharvey

I also want to take the opportunity, to all those interested, to invite you to the Pouchdb Channel from the official Couchdb Slack workspace. https://couchdb.slack.com/archives/C016TJAE7A4

I believe we can collaborate further there.

Thanks.

garbados

comment created time in 6 days

IssuesEvent

issue commentpouchdb/pouchdb

Implement partial_filter_selector?

Sorry about that, definitely need to reconsider this bot given the current activity levels

garbados

comment created time in 6 days

issue commentapache/couchdb-nano

Uncaught TypeError: axiosCookieJarSupport is not a function

I get the same error. I try to connect to the database like it says in your documentation: const nano = require('nano')('http://localhost:5984') but get the error mentioned above.

image

qxygene

comment created time in 6 days

issue commentpouchdb/pouchdb

Implement partial_filter_selector?

hey @daleharvey little help here, please.

garbados

comment created time in 6 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Intriguing, thanks for the idea @deepmouse

What do you think about this @garbados ?

deepmouse

comment created time in 6 days

issue closedpouchdb/pouchdb

AllDocs slow performances

<!-- Hello! 👋 Thank you for opening an issue on PouchDB. There is also a 3-Minute read, on how to format your issue: https://guides.github.com/features/mastering-markdown/ This template is optional, but it may help us fix your issue faster. -->

Issue

I'm use pouchdb sync data from couchbase, then query data via allDocs method is slow, it takes 6011ms to query 70k(documents count) documents, how to optimization query performance.

Info

  • Environment: browser
  • Platform: Chrome/FF/Safari/Edge
  • Adapter: IndexedDB
  • Server: couchbase server/sync gateway

Reproduce

use pouchdb sync 100k data from couchbase server, then query data use allDocs method

closed time in 6 days

dxwts

issue closedpouchdb/pouchdb

Implement partial_filter_selector?

<!-- Hello! 👋 Thank you for opening an issue on PouchDB. There is also a 3-Minute read, on how to format your issue: https://guides.github.com/features/mastering-markdown/ This template is optional, but it may help us fix your issue faster. -->

Issue

The partial_filter_selector property of Mango indexes is not handled by pouchdb-find. It should be!

Info

  • Environment: N/A
  • Platform: N/A
  • Adapter: Local (Remote CouchDB automatically handles partial indexes)
  • Server: N/A

Reproduce

Consider this example script, partial_index_test.js:

const assert = require('assert').strict

const PouchDB = require('pouchdb')
PouchDB.plugin(require('pouchdb-find'))

async function setup (db) {
  await db.createIndex({
    index: {
      fields: ['user_id'],
      partial_filter_selector: { type: 'post' },
      ddoc: 'queries',
      name: 'posts'
    }
  })
  await db.put({
    _id: 'user/test',
    type: 'user'
  })
  await db.put({
    _id: 'post/test',
    user_id: 'user/test',
    type: 'post',
    created_at: Date.now()
  })
  await db.put({
    _id: 'not-post/test',
    user_id: 'user/test',
    type: 'not-post',
    created_at: Date.now()
  })
}

const COUCH_URL = process.env.COUCH_URL
const DB_NAME = COUCH_URL ? `${COUCH_URL}/test` : '.test'
const db = new PouchDB(DB_NAME)

Promise.resolve().then(async () => {
  // create indexes, populate db
  await setup(db)
  // test the index
  const result = await db.find({
    selector: { user_id: 'user/test' },
    use_index: ['_design/queries', 'posts']
  })
  assert.equal(result.docs.length, 1)
  console.log('Output correct.')
}).catch((err) => {
  console.log(err)
}).then(async () => {
  await db.destroy()
})

This script instantiates a database with a Mango index and two documents. The index is created with a partial index that only returns documents of type "post". If this partial index is not applied, a similar document of type "not-post" will also be returned. As you can see...:

$ COUCH_URL="" node partial_index_test.js 
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

2 !== 1

    at /home/deus/code/scratch/db_find.js:45:10
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  generatedMessage: true,
  code: 'ERR_ASSERTION',
  actual: 2,
  expected: 1,
  operator: 'strictEqual'
}

Using a CouchDB installation by setting COUCH_URL generates the correct result:

$ COUCH_URL="http://[user:pass]@localhost:5984" node partial_index_test.js
Output correct.

closed time in 6 days

garbados

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Advantages:

  • users can sync app data between devices even when the central backend is down
  • same thing goes for if your cloud provider has an outage
  • using p2p network for data sync can reduce server costs
  • can write apps that persist data over IP without ever setting up, configuring or maintaining your own database server
deepmouse

comment created time in 7 days

issue commentpouchdb/pouchdb

Browser<->browser sync without a server

Other links relating to serverless data transfer over IP:

https://holochain.org/ https://hypercore-protocol.org/

deepmouse

comment created time in 7 days

issue openedpouchdb/pouchdb

Browser<->browser sync without a server

This seems highly doable, and might not even be that much work!

  • WebRTC allows two browsers to connect to each other and send data without a server in between
  • p2p networks that allow browsers to find each other in order to establish a WebRTC connection already exist: https://gun.eco/ (so no need to install a server if you don't want, you can use someone elses)
  • if direct connection between browsers fail, the data can be synced indirectly through peers on the p2p network

created time in 7 days

issue openedpouchdb/pouchdb

Electron with Pouchdb [Level Db ] insert error

<!-- Hello! 👋 Thank you for opening an issue on PouchDB. There is also a 3-Minute read, on how to format your issue: https://guides.github.com/features/mastering-markdown/ This template is optional, but it may help us fix your issue faster. -->

Issue

not a issue just a question on pouchdb with leveldb in electron full question link : https://stackoverflow.com/questions/67472024/electron-with-pouchdb-level-db-insert-error

Info

  • Environment: (Node.js - electron js)
  • Platform: (Chrome.)
  • Adapter: (LevelDB)

can any one help me with the answer for the question

created time in 7 days

pull request commentpouchdb/pouchdb

feat: support partial_filter_selector

@garbados Thanks for your work on this subject! Really appreciate it.

Yours tests are failing since you don't return the promise. Can you try to cherry-pick this commit https://github.com/Crash--/pouchdb/pull/1/commits/7dddcdfc2748e8eb5a19377b0b69292db5c18cb3 to see if the issue is fixed?

garbados

comment created time in 8 days

startedjhs/cqs

started time in 9 days