profile
viewpoint

dustin/seriesly 461

A time series database.

dustin/diff 7

Different strokes.

mschoch/AndroidGrocerySync 4

Grocery-Sync ported to Couchbase Mobile for Android

mschoch/Android-Couchbase 2

The android build for humans.

mschoch/blackfriday-text 2

A plain text formatter for the blackfriday markdown processor.

mschoch/bleve_sourcegraph_blog 2

Markdown and Examples for blog

mschoch/CouchAppCodaPlugin 2

A plug-in for the Coda editor to more easily support working with CouchApps

blugelabs/blugelabs.com 1

Bluge Labs Website

mschoch/android-async-http 1

An Asynchronous HTTP Library for Android

issue commentmattermost/mattermost-server

[Bleve] massive index size compared to indexed data size

I don't really have any context for this, and only saw it via a google alert, but it seems that indexes are created using the method bleve.New here:

https://github.com/mattermost/mattermost-server/blob/master/services/searchengine/bleveengine/bleve.go#L117

And this uses the default indexing scheme, which (due to backwards compatibility issues) is quite old. If the size of the index is important, we recommend creating indexes with the scorch type, and also one of the newer zap file format versions. Some more information can found here (although the latest zap is now 15): https://github.com/blevesearch/bleve/issues/1186#issuecomment-615335353

Obviously this may not be a simple change, as it potentially leads to backwards compatibility issues for this project as well...

XANi

comment created time in 7 days

push eventblugelabs/bluge

Michael Schuett

commit sha 0af8261a1a0b5d7c010223c1cfb90ce9e01593b4

Fix possible runtime panic on DecRef call The following error was hit when porting from bleve to bluge ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x17000ce] goroutine 16 [running]: github.com/blugelabs/bluge/index.(*closeOnLastRefCounter).DecRef(0xc0004b59e0, 0x10ca11a, 0x8000000000000000) /gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/segment_plugin.go:113 +0xae github.com/blugelabs/bluge/index.(*Snapshot).decRef(0xc0005a0780, 0x3, 0x8ffd3) /gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/snapshot.go:77 +0xb3 github.com/blugelabs/bluge/index.(*Snapshot).Close(...) /gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/snapshot.go:89 github.com/blugelabs/bluge/index.(*Writer).mergerLoop(0xc000069200, 0xc000606660, 0xc0000402a0) /gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/merge.go:75 +0x4c5 created by github.com/blugelabs/bluge/index.OpenWriter /gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/writer.go:131 +0x8cd ```

view details

Michael Schuett

commit sha fd4c59055c4f87fb3368ddacce0ddd1885dc5dba

Guard against possible panic on closer.Close() I placed some beautiful debug code inside this function and saw that it was possible for both err and closer to return nil. Add a guard around the close call.

view details

Michael Schuett

commit sha a9d780a476f097b97de443d4ac3e7ac5147afb20

Update and sort authors file

view details

Marty Schoch

commit sha c185b33846b7fbaea08659a5af2644ff21165a9b

fix handling of nil io.Closer in directory Load there were a few more places this needed to be handled correctly

view details

Marty Schoch

commit sha c304a6733af6b14a0e57acb8085f7e88bae98560

Merge pull request #34 from michaeljs1990/master Fix possible runtime panic on DecRef call

view details

push time in 7 days

PR merged blugelabs/bluge

Fix possible runtime panic on DecRef call

The following error was hit when porting from bleve to bluge

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x17000ce]

goroutine 16 [running]:
github.com/blugelabs/bluge/index.(*closeOnLastRefCounter).DecRef(0xc0004b59e0, 0x10ca11a, 0x8000000000000000)
	/gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/segment_plugin.go:113 +0xae
github.com/blugelabs/bluge/index.(*Snapshot).decRef(0xc0005a0780, 0x3, 0x8ffd3)
	/gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/snapshot.go:77 +0xb3
github.com/blugelabs/bluge/index.(*Snapshot).Close(...)
	/gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/snapshot.go:89
github.com/blugelabs/bluge/index.(*Writer).mergerLoop(0xc000069200, 0xc000606660, 0xc0000402a0)
	/gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/merge.go:75 +0x4c5
created by github.com/blugelabs/bluge/index.OpenWriter
	/gocode/pkg/mod/github.com/blugelabs/bluge@v0.1.3/index/writer.go:131 +0x8cd

It seems like might be possible that https://github.com/blugelabs/bluge/blob/fe1f453e701a72cb3ee01b13b8a5e5f6e2b0f6cc/index/writer.go#L523 could throw a panic also as I don't see any other place segmentWrapper is initialized. It looks like a path exists where err is nil and the returned closer also is.

I am using the bluge.InMemoryOnlyConfig.

+30 -10

7 comments

5 changed files

michaeljs1990

pr closed time in 7 days

PullRequestReviewEvent

pull request commentblugelabs/bluge

Fix possible runtime panic on DecRef call

@michaeljs1990 I pushed another commit, trying to address all the places Bluge internally uses the io.Closer returned from the Load method. If this looks good to you can we can merge.

michaeljs1990

comment created time in 8 days

push eventmichaeljs1990/bluge

Marty Schoch

commit sha c185b33846b7fbaea08659a5af2644ff21165a9b

fix handling of nil io.Closer in directory Load there were a few more places this needed to be handled correctly

view details

push time in 8 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 4c379a21bd1a6550138d689cb53178014899bae7

make video bigger

view details

push time in 8 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha f84ef91a51af7d3c65d0a49c080990fb925ee2a2

try updated embed

view details

push time in 8 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 6671e85255189f6fcabac0168a84fb1e0001b87d

add video introduction to bluge blog post

view details

push time in 8 days

create barnchblugelabs/bluge_examples

branch : master

created branch time in 9 days

created repositoryblugelabs/bluge_examples

A collection of examples illustrating the use of Bluge

created time in 9 days

issue commentblugelabs/bluge

Export fields in BaseSearch?

I don't think there is any hard objection here. The BaseSearch is a new construct, as Bleve only supported TopN functionality, and Bluge had as a goal supporting AllMatches functionality which returned results accessible via the same API.

You mentioned "packages adding searchers", so I just want to clarify (or possibly improve) some terminology. I tend to use the word Searcher to describe something implementing the search.Searcher interface.

Making change you propose would aid in adding something other than TopN or AllMatches, I'm not sure what to best name that.

Can I ask what extension you have in mind here?

michaeljs1990

comment created time in 10 days

pull request commentblugelabs/bluge

Fix possible runtime panic on DecRef call

Yeah, I think you've persuaded me. Since the Directory is an important new way to extend Bluge, we want to encourage other implementations, and that means consumers have to be prepared to deal with a nil closer.

Tomorrow I'll do a deeper review, as there are a few more places not currently captured in this PR.

michaeljs1990

comment created time in 13 days

issue closedblevesearch/bleve

Documentation says simple analyzer uses unicode tokenizer

According to the documentation, the simple analyzer uses the unicode tokenizer. It doesn't. It uses the letter tokenizer.

closed time in 13 days

nkovacs

issue commentblevesearch/bleve

Documentation says simple analyzer uses unicode tokenizer

The fix has been merged.

nkovacs

comment created time in 13 days

pull request commentblugelabs/bluge

Fix possible runtime panic on DecRef call

So, looking at this a bit more, I realize there are several paths to be considered (every place that calls Load() on a directory). This include snapshots, as well as segments, and also places like the OfflineWriter.

So now I'm wondering if we should update the contract of the Directory interface to always return a non-nil closer. And we can simply have a singleton noopCloser that does nothing that everyone can return if needed. That will reduce the burden on all consumers of the Directory, as they can simply call Close() without additional checks.

michaeljs1990

comment created time in 13 days

push eventblevesearch/blevesearch.github.io

Blevesearch Site Auto Publisher

commit sha 879b2a5c4d43e5744c7ebf454b78909b959ddbd9

Built from commit f103022435f7519f1fbcb13d33a06d0df6bc138c

view details

push time in 13 days

push eventblevesearch/blevesearch.github.io-hugo

Marty Schoch

commit sha d56c83ad1dbc29253eac3f5d83b8b372cbdb5fc1

Update Analyzers Documentation The description of the simple analyzer incorrectly references the Unicode tokenizer. This updates the description to refer to the Letter tokenizer.

view details

Marty Schoch

commit sha f103022435f7519f1fbcb13d33a06d0df6bc138c

Merge pull request #23 from blevesearch/fix-simple-analyzer-docs Update Analyzers Documentation

view details

push time in 13 days

delete branch blevesearch/blevesearch.github.io-hugo

delete branch : fix-simple-analyzer-docs

delete time in 13 days

PR merged blevesearch/blevesearch.github.io-hugo

Reviewers
Update Analyzers Documentation

The description of the simple analyzer incorrectly references the Unicode tokenizer. This updates the description to refer to the Letter tokenizer.

+1 -1

1 comment

1 changed file

mschoch

pr closed time in 13 days

issue commentblugelabs/bluge

How to get a single field value without iterating all of them

So, the reasoning behind this is that all of the stored fields are encoded as a single compressed entry. The cost to retrieve a single value is essentially the cost to retrieve any value, thus the API takes the form that it does. This is similar to Lucene, which (last time I checked) had a similar API.

We could offer an API to access a single value, but depending on how it was used, it could be even more expensive.

michaeljs1990

comment created time in 14 days

issue openedblugelabs/bluge

should Writer's Reader() method return err?

Currently it returns (*Reader, error), but it the error is always nil.

created time in 14 days

pull request commentblugelabs/bluge

Fix possible runtime panic on DecRef call

Thanks for reporting this. I think this results from the fact that when we load segments from an in-memory directory, there is no meaningful close operation, and we return nil. Then as you found, elsewhere in the code we expect the closer to be non-nill.

I think this fix is good, but want to review it a bit more before merging.

In the meantime, can you push another commit, adding an appropriate entry to the AUTHORS file for copyright purposes?

michaeljs1990

comment created time in 14 days

PullRequestReviewEvent

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha e2f25e6fe6c089aaeb6375afc0b44b1f02548a81

add community links

view details

push time in 15 days

PullRequestReviewEvent

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 249607932785ef1adb422db95534619bcef9f7a9

site search blog post

view details

push time in 15 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 97c86e9fb27866b5e4a9c501e71d268c9554dbc8

remove search duration from message duration is unreasonably low in serverless env as it does not include startup

view details

push time in 15 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 823aeb277778f8339451d13dd4f70b256040745c

clean up index code

view details

push time in 15 days

issue commentblevesearch/bleve

2 questions.

  1. It's difficult to say for sure. The largest deployments I'm aware of are not owned by the companies I directly work with, but by their customers, so the information shared is limited. I believe Couchbase does testing at 100 million items, but that is in aggregate split across a few separate bleve index partitions.

  2. Production ready means different things to different people. There are already several companies I've worked with using Bleve in production. Here are a few things you should think about:

  • Bleve releases today are a best effort community release. It means we do the best we can, and we think the releases are good. But, this isn't enough for most of the companies I've worked with. They all do their own testing internally before adopting a new release.
  • You need to think about supporting indexes over time. Over time bleve has evolved from the upsidedown index format to scorch. And scorch has multiple file format versions within it. Each of these came with significant benefits, but rebuilding indexes has a cost, so many users need to continue to support older indexes.
  • You should think about how you plan to support your use of bleve. Will you be digging into the code, maintaining a fork and proposing changes upstream? Or will you prefer to pay someone to help you? Your earlier questions suggest you don't plan on using scorch, and instead may use some other k/v store, and that may significantly limit the community support you receive as well as the paid options available to you.
  1. There are too many unknowns for me to be able to directly answer this. The instances I'm aware of in the 100mil document range are all disk based, but you mentioned in-memory, so I have no experience to draw on there. Because they are disk-based, the index itself is largely mmap'd meaning isn't part of the heap. That said, Go runtime and linux kernel accounting change over time and there have been issues from time to time. You probably should expect to face the usual challenges associated with such processes, tuning swap, over-commit, etc. Finally, you offer some math about bytes per item, but again earlier you mentioned using the upsidedown index type with your own key/value store. This uses an extremely wasteful encoding that most likely goes way past your budge.

In summary, I'd suggest you start with a smaller-scale test to see if the numbers are even close. Also, if you go with a solution using the upsidedown index type and your own key/value store, there will be very little support.

gitmko0

comment created time in 16 days

issue commentblevesearch/bleve

2 questions.

  1. Bleve has a numeric type that is designed to support numeric range queries (making a poor choice for IDs where you typically only perform exact matches). Further, while the type encodes 64-bits of information, all of the higher-level code interprets the value as a float64 meaning you cannot represent all 2^64 integer values. None of the bleve types support > 64-bits.

Overhauling Bleve's support for numeric values (most likely using something like BKD-tree as lucene does) is on the roadmap.

  1. Scorch is not the default in 1.x and we have not taken away the ability to use in-memory indexes. That said, the index created by the NewMemOnly method is using some pretty old code. The alternative that most people use is to invoke NewUsing and configure a KV-store of your choosing that doesn't persist to disk.

An in-memory index using scorch technology is available in the bluge developer preview, but it has not been back-ported to bleve.

gitmko0

comment created time in 16 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 6b4c07de586c493c79154811ba6701fd73a0101f

don't index the search page itself

view details

push time in 16 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 99c2e4c7bfc279e8f46ad16d3490ef5cfa4dcd0e

attempt to add search

view details

push time in 16 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 07e058dd860ab87385c0a0ad672486a7449e73a4

fixing page meta before adding search

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 6424cf5335d3dc0036bd080cf884fec4c7f35e29

syncing to current

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 5c7adeedfd7b84236ef8b19348261a4167766d7e

at least change it to match current repo

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha b44c386feafe1525122565d607878d294f4e5ba9

add it back

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 3642f48cf5247d048a82d5a91ea94288329ae229

try removing line that isn't applicable

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha b32846dc8085b73249328df94e229a6b5113283a

fix page meta

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 29aa4445bb475e56d3bf18945de97750784bfd77

replace dot as well

view details

push time in 16 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 0b5a05212601b06ac430fc988efd3ca45dd53213

add search to menu

view details

push time in 17 days

pull request commentblevesearch/blevesearch.github.io-hugo

Update Analyzers Documentation

Addresses: https://github.com/blevesearch/bleve/issues/1486

mschoch

comment created time in 17 days

push eventblevesearch/blevesearch.github.io

Blevesearch Site Auto Publisher

commit sha 79b96ed0884dff9d7f425280fade64412ee4110c

Built from commit d56c83ad1dbc29253eac3f5d83b8b372cbdb5fc1

view details

push time in 17 days

issue commentblevesearch/bleve

Documentation says simple analyzer uses unicode tokenizer

@nkovacs proposed fix: https://github.com/blevesearch/blevesearch.github.io-hugo/pull/23

nkovacs

comment created time in 17 days

PR opened blevesearch/blevesearch.github.io-hugo

Update Analyzers Documentation

The description of the simple analyzer incorrectly references the Unicode tokenizer. This updates the description to refer to the Letter tokenizer.

+1 -1

0 comment

1 changed file

pr created time in 17 days

push eventblugelabs/blugelabs.com

Marty Schoch

commit sha 2ea526812a36f63d7ba313a919912f465395d1f7

add Michael Schuett as a sponsor

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 3109dc04e057dfebc459e9b13d64322714586f9d

assign it

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 03be54f906918837014a125b446dcaaa3079349c

simplify more

view details

Marty Schoch

commit sha 9e0eef544ea977c32645b3b6bcf1e880e0245209

moremore

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha ddbfa1cadc5cb29f5475f09791ebc203208227b6

log toggle

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha b268f350f604357d79237dacda69e332a0fabc9d

oops

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 10894093bfc46bbc55e9fe2479c19f0da64b0cdc

switch to regexp

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 4481dfd0c136c1aabfb4ee3cfc8ab1f84617a7a1

try to fix scores

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha b9646f55ed59dcba49a4f8d6b55eae69579feb3d

make title link

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha a1c872a7b431cebff420b55d638dee8c7f5414aa

work on pagination buttons

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 3786191c32338837ce280480e91cd7c1bcbfb3ad

more

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha e854d410f57aa4ba55488e85a6a0eb02ac4075e0

try

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 9b60aab5a8e4693aea696832f94a236eba61cd4a

oops

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 2ee816b7b49e3bca55f9a28a2c96da3387575960

fix agg styling

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 0d1f554d820ecd11909d63894bc340bc4994b359

adjust mesage, try aggs

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 4e0257fb0b21cf3948fafe138f9e50044399cadc

yes

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha cf2f8e152a88bbf0a392e602e346fd94129c8e74

another go

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha c19c56890f902e812322ded985a16236462734bc

try a row

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha f64ac92645313774306bc34e1791ec8c5b902d72

again

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 753424604a33b87a86ba24673410027ae2a5fc9b

tighten

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha c2f088fbb816055048d196b61f6e499eb07eb055

adjust it

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha fe48ea976db39d628bc2f64be3c25518db75f164

fix message

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha ef660a27f22a7616c29c58c61b682426b5cd02b1

mark brighter yellow

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 51d9302ca3bbdd91e2c9f799494e2782b080a797

tripple stash the content

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 4e5a7824bdfc0d4aece3a8bac68f9cae16f890da

store term vectors

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 5f685475f2379dbec5c4a8cc02ed68522163f7bb

include locations

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha e5c6b964df50bfa6675e6a598e22e9cf88b523c9

fix results

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 8c0cca85fd9f4a2c82c36a13aff509a4da68c43d

reformat search

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha cdd888c3f1ff72c00dc4d1412acef0b602f3ad3d

fix layout

view details

push time in 17 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha fc72431510f7a79704691c1ddf3a64847d5c4199

try that

view details

push time in 18 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha ed7a1a862c453a920338b6143e182cb0ee40b819

move inside

view details

push time in 18 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha c491e6a1e602ff2e4baf956d4b9e12782b1ee647

fix form

view details

push time in 18 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha b64a0aa7669c12a1d647b06deb91f0f3b6e3e0f9

do it manually

view details

push time in 18 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha e5037602a6f4dd2b097a557c2eeb8cca1574ae3c

add bulma

view details

push time in 18 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 05d7e44ecf6c9d4c4e33b53636c714b98066555e

set content type

view details

push time in 19 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 9cd159ae9aab8429944d88abfa4a435dff7a2003

fill out search

view details

push time in 19 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 0cfbe762b08d7e392c28027d42cb8fb9a55e1e0c

ffff

view details

push time in 19 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha b8b8ca06d8c8f6c842999d76aa99ae2a21cc96c2

fml

view details

push time in 19 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 4c173294f4f6b321dbc643ab8bd46c6e25ef8394

update

view details

push time in 19 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 2049188d0004a5f2eea4f3d1de13c0a7f145a239

add search page

view details

push time in 19 days

PullRequestReviewEvent

issue openedblevesearch/bleve

bad combination of default true (in JSON unmarshal) and omitempty (marshal) for docvalues_dynamic

During JSON Unmarshaling of an IndexMappingImpl we default DocValuesDynamic to the package level setting (which itself defaults to true). This causes any incoming JSON mapping which omits the DocValuesDynamic setting, to default to true.

https://github.com/blevesearch/bleve/blob/f3fe7cc2de465ba037105df7f4d756603e2bdf7c/mapping/index.go#L222

But, that same field has it's JSON struct tag declare omitempty:

https://github.com/blevesearch/bleve/blob/f3fe7cc2de465ba037105df7f4d756603e2bdf7c/mapping/index.go#L53

This is problematic because the following sequence of events can cause the value to flip unintentionally:

  1. User creates mapping with JSON docvalues_dynamic set to false.
  2. User retrieves the JSON for the purpose of editing, here the docvalues_dynamic is now missing. It is omitted because the value is false, which is the zero value for boolean.
  3. User makes some other unrelated change and submits the JSON with docvalues_dynamic missing. This is now interpreted by the Unmarshal method as needing a default value substituted, which sets it to true.

created time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 18c33f127456a3299068ad1609b6003eef1de31c

newer bluge

view details

push time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 1a59eba556a97c65e4443b9c46927a21fcb00bee

remove failing cmd

view details

push time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha cd455fb0b1e1f155fa4bd4cead195602fd7e21bd

try set GOBIN

view details

Marty Schoch

commit sha facbf46b1f6945d51be57d03f4c04dbb1c441366

dont try that

view details

Marty Schoch

commit sha c3deb3a310f92a2787e8544540090abc6f2415de

try to work around

view details

push time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 0c6646d4d1b68e408493470c8175c97a325bb37f

look for binary

view details

push time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 71ebc5b64ace661abddc4271110b85f1b314bffe

remove reference to local dir

view details

push time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 49af372f26f50ffadbb17c94f3612be129b827ff

add build target to Makefile

view details

push time in 20 days

push eventmschoch/aws-lambda-go-example

Marty Schoch

commit sha 619e4b50ceadb2ead789cc9bedf4352f57afa3f2

mix site with functions

view details

push time in 20 days

release blugelabs/bluge

v0.1.3

released time in 21 days

created tagblugelabs/bluge

tagv0.1.3

indexing library for Go

created time in 21 days

issue commentblevesearch/bleve

Set internal row only if changed

Yeah I don't know what to suggest. The New and Open methods have a built-in assumption that some meta-data will be stored on disk at the specified path. The new case will just ignore it if the path is empty, because we reuse this path for our in-memory indexes. You have found this and started using the New method for both creating new indexes, as well as opening existing ones.

As the new method expects to only be used when creating a new index, it always attempts to store the index mapping. The open method is closer to what you want, but does not skip the file operations if the path is empty. And it cannot meaningfully skip those, as we use the loaded meta-data to know which kvstore to use.

It seems your use-case is outside the expectations of the bleve API. I would be opposed to introducing an option that caused the new method to skip writing the index mapping, as the method is only intended to be used for new indexes.

I think the best option I see is to introduce an option overrideMeta honored only by the openIndexUsing method. If you set that to some value, we skip the call to openIndexMeta (which is what fails when path is empty), and would instead initialize the important values of the *indexMeta directly from the provided config (things like IndexType and Storage are used later in the method).

I prefer this solution because it keeps the intended usage of new vs open intact. Would something like this work for you?

phifty

comment created time in 22 days

release blevesearch/bleve

v1.0.12

released time in 22 days

created tagblevesearch/bleve

tagv1.0.12

A modern text indexing library for go

created time in 22 days

delete branch blevesearch/bleve

delete branch : prepare-v1.0.12

delete time in 22 days

more