profile
viewpoint
Goutham Veeramachaneni gouthamve Grafana Labs Berlin, Germany @prometheus & @cortexproject Maintainer. SWE @grafana

cortexproject/cortex 4446

A horizontally scalable, highly available, multi-tenant, long term Prometheus.

gouthamve/agni 5

Experimental LTS for Prometheus

coderplex-org/cloud-month 2

Repo for Cloud Month. Event description, slides and resouces on cloud computing

gouthamve/alertmanager-irc-relay 2

Send Prometheus Alerts to IRC using Webhooks

gouthamve/client_golang 1

Prometheus instrumentation library for Go applications

gouthamve/ColorSim 1

Finds the CIEDE2000. Implemented to learn Dropwizard

issue commentopen-telemetry/opentelemetry-collector-contrib

Prometheus Exporter - Gauge counter metrics dropped with error 'failed to translate metric'

you must clear/delete the values recorded for the instrument

Which SDK are you using? I cannot see any method to do the same in the Golang SDK.

rahuls2

comment created time in 3 days

issue commentopen-telemetry/opentelemetry-collector-contrib

Prometheus Exporter - Gauge counter metrics dropped with error 'failed to translate metric'

Hi, I am new to the project and I am trying to reproduce this. I have a few questions:

Use an application to send opentelemetry gauge type metrics to otelcollector once every 30 seconds.

Does this mean setting the collector period to 30secs? Like this:

	metricClient := otlpmetricgrpc.NewClient(
		otlpmetricgrpc.WithInsecure(),
		otlpmetricgrpc.WithEndpoint(otelAgentAddr))
	metricExp, err := otlpmetric.New(ctx, metricClient)

        pusher := controller.New(
		processor.NewFactory(
			simple.NewWithExactDistribution(),
			metricExp,
		),
		controller.WithExporter(metricExp),
		controller.WithCollectPeriod(30*time.Second),
	)

For an exact replica, some of the instruments must not have any values recorded every once in a while.

What do you mean by not have any values recorded every once in a while? I am recording the value for the gauge every 120s, like this:

	newGauge := metric.Must(meter).
		NewInt64UpDownCounter(
			"repro_issue/elastic_stuff",
			metric.WithDescription("This is me desperately trying to reproduce the issue"),
		)

	for {
     		time.Sleep(time.Duration(120) * time.Second)
		meter.RecordBatch(
			ctx,
			commonLabels,
			newGauge.Measurement(10),
		)
	}

I am unable to reproduce this issue with this setup though.

rahuls2

comment created time in 6 days

pull request commentprometheus/prometheus

tsdb/wal: add support for saving and restoring read segments

Also, this means we'll be resending data on restarts. Not sure if it breaks some storages. @tomwilkie / @cstyan / @csmarchbanks Can you confirm that its okay to resend on restarts?

rfratto

comment created time in 7 days

pull request commentprometheus/prometheus

tsdb/wal: add support for saving and restoring read segments

I'm seeing stuff like "LastCheckpointedSegment" and the first thought goes to WAL Checkpointing. Can we rename this new file to something else? RWMarker or something? Don't have good ideas though.

rfratto

comment created time in 7 days

push eventgouthamve/agent

Goutham Veeramachaneni

commit sha d26e86d3d580c8a5bd665f36d8b3fd278828ead7

Fix the tests Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 8 days

PR opened grafana/agent

Remove WAL on startup

Agent / Remote Write code currently doesn't send the samples from the WAL on startup at all. We only send the new samples so replaying the WAL doesn't give us much.

It does let us load series references from the WAL but we save much more by not reading the WAL at all.

See: https://github.com/prometheus/prometheus/issues/9848 and https://github.com/prometheus/prometheus/pull/8918

PR Checklist

  • [ ] CHANGELOG updated
  • [ ] Documentation added
  • [ ] Tests updated
+6 -0

0 comment

1 changed file

pr created time in 8 days

create barnchgouthamve/agent

branch : nuke-wal-on-startup

created branch time in 8 days

fork gouthamve/agent

Prometheus Metrics, Loki Logs, and Tempo Traces, optimized for Grafana Cloud.

https://grafana.com/docs/agent/

fork in 8 days

issue commentprometheus/prometheus

Prometheus Agent: consider never moving samples to checkpoint

I wouldn't say its not providing any benefits. It is, when the remote is down.

Having said that, it doesn't help when the agent restarts. We basically restart onto a clean slate, and can save a lot of time during restarts by just deleting the WAL.

rfratto

comment created time in 9 days

issue openedgrafana/agent

WAL cleanup panics if the WAL is not yet loaded

ts=2021-11-23T12:26:09.634937274Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
ts=2021-11-23T12:26:09.63557965Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
ts=2021-11-23T12:26:09Z level=info caller=traces/traces.go:123 msg="Traces Logger Initialized" component=traces
ts=2021-11-23T12:26:09.648577389Z caller=server.go:77 level=info msg="server configuration changed, restarting server"
ts=2021-11-23T12:26:09.649497502Z caller=gokit.go:72 level=info http=[::]:3090 grpc=[::]:9095 msg="server listening on addresses"
ts=2021-11-23T12:26:09.786774813Z caller=wal.go:182 level=info agent=prometheus instance=14d8e3c77664fe89fcfcb980c9ba0874 msg="replaying WAL, this may take a while" dir=/var/lib/agent/data/14d8e3c77664fe89fcfcb980c9ba0874/wal
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1a293f0] 
goroutine 253 [running]:
github.com/grafana/agent/pkg/metrics/instance.(*Instance).StorageDirectory(0xc000889e40, 0xc08d627830, 0xc00063fd78)
/src/agent/pkg/metrics/instance/instance.go:559 +0x30
github.com/grafana/agent/pkg/metrics.(*WALCleaner).getManagedStorage(0xc00013e1e0, 0xc08d627830, 0x1)
/src/agent/pkg/metrics/cleaner.go:154 +0x9d
github.com/grafana/agent/pkg/metrics.(*WALCleaner).cleanup(0xc00013e1e0)
/src/agent/pkg/metrics/cleaner.go:248 +0xb7
github.com/grafana/agent/pkg/metrics.(*WALCleaner).run(0xc00013e1e0)
/src/agent/pkg/metrics/cleaner.go:237 +0x98
created by github.com/grafana/agent/pkg/metrics.NewWALCleaner
/src/agent/pkg/metrics/cleaner.go:145 +0x205

Here the WAL loading takes longer than 30mins, at which point the WALCleaner runs. But because WAL is nil, it panics. We should fix that.

created time in 9 days

pull request commentgrafana/agent

Setup CD for agent

So this works! And it directly commits to master and doesn't open a PR. Which is what we want for the dev rollouts.

gouthamve

comment created time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 3564a9f3e5033015590e8bd7b9ce92e80893f23f

Finally make it work! Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 2ea2bfe5c3d32bf6dd5e3906da787864b1ba218d

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 013b9f465f1d7916a1f2b08fe8a2047ee008f64e

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 5e1430868267dcd7d18533d5feedfd33fbd096a8

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 369adf6a380e4483aab2a5078ab830931b549ec7

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 5b1ae6d94f957b726357e2d80e678eae609b39ef

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha a0f20804c4d663d4378b7477191df3702ba91a6a

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 5b3eda839c4a931d4ca842e0d1e409ae6935a32e

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha ce012406f9a91789517e8b49dfdebdbc3a996577

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 12803bddee14802c9781a39bd6833bb44aa2f772

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 37bee92b355cd91c3adb803602bac888e181c693

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 9f0399c1d3c9ed8c687cf3be5ed86b98ac61d662

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 593026aa71704190f5ae33ba0db4eeaaa535e229

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha d9e53c4dc62089c2a6470b0b9a9e623528a2315a

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 8a8e970bac43075a33fa190c9701be6fa6c40e33

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 6053f3ebee659e3e1fc7858e4a6a8d1e2e83f6bc

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha f855e32b4b7b97c5e2cc95d9c74b60869d85c831

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha f4bbb8e727c4e4135d74fc3d825707e8088c98ad

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

push eventgrafana/agent

Goutham Veeramachaneni

commit sha 251e5c9653cc19c92487ad0471553228595329a8

Testing commit. Will be removed Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

view details

push time in 13 days

more