profile
viewpoint

Ask questionsWhile under heavy load getting error panic

Describe the bug Server will not start. Getting: Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: panic: runtime error: index out of range [32] with length 32 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: goroutine 192 [running]: Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/klauspost/compress/zstd.(*fseEncoder).writeCount(0xc00d674000, 0xc0141ae000, 0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/vendor/github.com/klauspost/compress/zstd/fse_encoder.g Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/klauspost/compress/zstd.(*blockEnc).encode(0xc0002a0ee0, 0xc0002a0ee0, 0xc0052 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/vendor/github.com/klauspost/compress/zstd/blockenc.go:6 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/klauspost/compress/zstd.(*Encoder).EncodeAll(0xc0001cad80, 0xc0052e5d6e, 0x0, Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/vendor/github.com/klauspost/compress/zstd/encoder.go:47 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding/zstd.CompressLevel(0xc00a990000, Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/encoding/zstd/zstd_pure.go:43 +0x68 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/encoding.CompressZSTDLevel(0xc00a990000, 0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/encoding/compress.go:16 +0x98 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*inmemoryBlock).marshalData(0xc0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/encoding.go:241 +0x5ec Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*inmemoryBlock).MarshalSortedDat Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/encoding.go:166 +0x98 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*blockStreamWriter).WriteBlock(0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/block_stream_writer.go:167 +0x6c Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*blockStreamMerger).flushIB(0xc0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/merge.go:193 +0x488 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*blockStreamMerger).Merge(0xc000 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/merge.go:134 +0xf0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.mergeBlockStreams(0xc00df89cd8, 0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/merge.go:35 +0x11c Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*Table).mergeParts(0xc0001ca900, Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/table.go:774 +0x4fc Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*Table).mergeExistingParts(0xc00 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/table.go:653 +0x120 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*Table).partMerger(0xc0001ca900, Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/table.go:667 +0x90 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*Table).startPartMergers.func1(0 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/table.go:635 +0x30 Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: created by github.com/VictoriaMetrics/VictoriaMetrics/lib/mergeset.(*Table).startPartMerg Oct 21 04:57:16 linux-infrastructure-chicagodbaas-influx-10.novalocal victoria-metrics-pure-1.28.0.ppc64le[54144]: /root/vendor_repo/VictoriaMetrics/lib/mergeset/table.go:634 +0x68

To Reproduce Unknown how to reproduce. Instance is under heavy load and after about 1 week of running error started to occur.

Expected behavior No error to occur

Screenshots If applicable, add screenshots to help explain your problem.

Version 1.28.0 (ppc64le pure compiled version)

Additional context image

image

image

image

image

VictoriaMetrics/VictoriaMetrics

Answer questions klauspost

Yes. Definitely looks like something that shouldn't happen. I will take a look.

useful!
source:https://uonfu.com/
Github User Rank List