Ask questionsPrometheus drop-in replacement in grafana not working correctly (last rows - NaN)

I try replace prometheus with victoriametrics in grafana and I observe strange behavior of the VictoriaMetrics. It is clearly visible on node_exporter dashboard in grafana. From response from VM, the last several values are NaN, after a while these values appear. Elements that use the last value (current value) - do not work correctly. This behavior is not seen with standard prometheus.

{ "xhrStatus": "complete", "request": { "method": "GET", "url": "api/datasources/proxy/2/api/v1/query_range?query=node_memory_MemFree_bytes%7Binstance%3D~%22testserver.local%3A9100%22%2Cjob%3D~%22test-prometheus-servers%22%7D&start=1559580404&end=1559580440&step=4&timeout=1s" }, "response": { "status": "success", "data": { "resultType": "matrix", "result": [ { "metric": { "__name__": "node_memory_MemFree_bytes", "instance": "testserver.local:9100", "job": "test-prometheus-servers" }, "values": [ [ 1559580404, "5702692864" ], [ 1559580408, "5702909952" ], [ 1559580412, "5702909952" ], [ 1559580416, "5703041024" ], [ 1559580420, "5703041024" ], [ 1559580424, "5703041024" ], [ 1559580428, "5703041024" ], [ 1559580432, "NaN" ], [ 1559580436, "NaN" ], [ 1559580440, "NaN" ] ] } ] } } }


# victoria-metrics -version victoria-metrics-20190603-091324-heads-master-0-gda45a20 # grafana-server -v Version 5.4.2 (commit: d812109, branch: HEAD) # prometheus --version prometheus, version 2.10.0 (branch: HEAD, revision: d20e84d0fb64aff2f62a977adc8cfb656da4e286)


Answer questions rumanzo

Hi! I attach prometheus and victoriametrics logs and configs and env's (args).

I made several requests to VictoriaMetrics and I have a suspicion that this behavior is due to the fact that Grafana use the browser time in query (absolute, not relative like "now"), and the time on the server is not synchronized. VictoriaMetrics gives series of NaN values if end time in request from the future. I check clock skew from from workstation and server, and diff ~15s, and if I curl api with &end=now - all ok, no NaN, and if I change time parameter in grafana to now-20s - all ok. Tomorrow I synchronize the time on the server, and probably the problem will go away, but I’m not sure if this is the correct behavior.

victoriametrics.log prometheus.log prometheus.yml.txt victoriametrics.env.txt prometheus.env.txt

Github User Rank List