profile
viewpoint

LuboVarga/gatling-amqp 2

Gatling AMQP support

LuboVarga/json4s-path 1

JsonPath for Json4s

LuboVarga/client_java 0

Prometheus instrumentation library for JVM applications

LuboVarga/dashboards 0

grafana dashboards for grafana.net

LuboVarga/docs 0

Prometheus documentation: content and static site generator

LuboVarga/erlang_guidelines 0

Inaka's Erlang Coding Guidelines

LuboVarga/grafana 0

The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More

LuboVarga/http-rpc-client 0

RPC over HTTP

LuboVarga/Hystrix 0

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.

LuboVarga/jackson-dataformats-binary 0

Uber-project for standard Jackson binary format backends

issue openedjenkinsci/prometheus-plugin

"Successful build count" and "Failed build count" counters are decreasing some times

Steps to reproduce

  1. Create a test job which runs regularly.
  2. Check the success and failed build count counters graph.

Expected behaviour

The successful and failed counters must never decrease.

Actual behavior

The successful and failed counters go up and down.

Note

Possibly this could be related to the Jenkins "discard old builds" strategy (log rotation)

Server configuration

Operating system: Ubuntu 18.04 Jenkins Version: 2.249.1 Plugin Version: 2.0.7

Notify

@markyjackson-taulia

created time in a day

issue commentjenkinsci/prometheus-plugin

Build success and failure count should be monotonically increasing number

@marceloavilaoliveira yeah, lets open a new issue and track from there

palamvmw

comment created time in 2 days

issue commentjenkinsci/prometheus-plugin

Build success and failure count should be monotonically increasing number

Does it count the successful/failed builds using the backlog? I thought it would increment the counters every time a build is finished.

A counter must never decrease. Let me know if you want that I open a new issue. Thanks a lot.

palamvmw

comment created time in 2 days

issue commentjenkinsci/prometheus-plugin

Build success and failure count should be monotonically increasing number

Hi @markyjackson-taulia

I'm having the same issue. The "default_jenkins_builds_success_build_count" and "default_jenkins_builds_failed_build_count" are decreasing, as you can see in the following example:

counter_issue

I'm wondering if this could be related to the Jenkins "discard old builds" strategy (log rotation), is it possible?

Operating system: Ubuntu 18.04 Jenkins Version: 2.249.1 Plugin Version: 2.0.7

This could very well be related to that

palamvmw

comment created time in 2 days

issue commentjenkinsci/prometheus-plugin

Build success and failure count should be monotonically increasing number

Hi @markyjackson-taulia

I'm having the same issue. The "default_jenkins_builds_success_build_count" and "default_jenkins_builds_failed_build_count" are decreasing, as you can see in the following example:

counter_issue

I'm wondering if this could be related to the Jenkins "discard old builds" strategy (log rotation), is it possible?

Operating system: Ubuntu 18.04 Jenkins Version: 2.249.1 Plugin Version: 2.0.7

palamvmw

comment created time in 2 days

pull request commentjenkinsci/prometheus-plugin

chore(ci): change to use Azure Container Instances for build

hmm, okay. If you have the time would you be able to elaborate why? (genuinely curious)

h1dden-da3m0n

comment created time in 13 days

pull request commentjenkinsci/prometheus-plugin

chore(ci): change to use Azure Container Instances for build

We cannot use this with the current CI infra that these builds run on

h1dden-da3m0n

comment created time in 13 days

PR opened jenkinsci/prometheus-plugin

chore(ci): change to use Azure Container Instances for build

Fixes #184(comment)

Up to you if you think running the builds of this plugin in containers brings a benefit, but it should prevent errors like No space left on device

Changes proposed

  • use Azure Container Instances for build

Checklist

  • [ ] Includes tests covering the new functionality?
  • [x] Ready for review
  • [x] Follows CONTRIBUTING rules

Notify

@markyjackson-taulia

+1 -1

0 comment

1 changed file

pr created time in 13 days

pull request commentjenkinsci/prometheus-plugin

change log level to debug

okay that was fast, thank you very much @markyjackson-taulia :+1: :tada:

yimgzz

comment created time in 13 days

push eventjenkinsci/prometheus-plugin

Sergey.Kalashnikov

commit sha 02699d01e420f6f58985897b0d36125251946ef9

change log level to debug

view details

Marky Jackson

commit sha c8d4af47013b710394334967a84c62dbb82a42e3

Merge pull request #184 from yimgzz/master change log level to debug

view details

push time in 13 days

PR merged jenkinsci/prometheus-plugin

change log level to debug

Fixes https://github.com/jenkinsci/prometheus-plugin/pull/167#issuecomment-712780595

Changes proposed

https://github.com/jenkinsci/prometheus-plugin/pull/167#issuecomment-712780595

Checklist

  • [ no need ] Includes tests covering the new functionality?
  • [+] Ready for review
  • [+] Follows CONTRIBUTING rules

Notify

@markyjackson-taulia

+2 -2

3 comments

1 changed file

yimgzz

pr closed time in 13 days

pull request commentjenkinsci/prometheus-plugin

change log level to debug

@markyjackson-taulia sorry to ping you but could you please have another look at this one, it really just touches two lines and the CI failed due to a completely unrelated error (the agent the job was scheduled on had a full HDD/SDD)

NOTE: IMHO change the Jenkinsfile of this plugin to run in a container and it should prevent any future No space left on device errors.

buildPlugin(useAci: true)
yimgzz

comment created time in 13 days

issue commentjenkinsci/prometheus-plugin

Prometheus API return blank page

Hi, We are facing this issue also in version 2.0.8

JingangLi

comment created time in 16 days

pull request commentjenkinsci/prometheus-plugin

change log level to debug

Ci is not passing. Please review

Since I noticed the Log spam today of this plugin, and see this PR being blocked by a faild CI, may I point at the CI msg: /home/jenkins/workspace/Plugins: No space left on device this has nothing to do with the change of this PR, just that the CI ran out of space while it tried to build this change.

yimgzz

comment created time in 21 days

issue commentjenkinsci/prometheus-plugin

Names & Labels for jenkins_node_*_builds Metrics

@Eric-Fontana-Bose this is better suited to adding labels in Prometheus.yml, that way you can template based on team/env. As in

- label_values
    env : dev
  targets
    - jenkins_dev.domain.com
- label_values
    env: prd
  targets
    - jenkins_prd.domain.com

Then your query becomes

jenkins_metric{env="prd"}

(Sry for formatting, on mobile)

james-m-tubbs

comment created time in 21 days

issue commentjenkinsci/prometheus-plugin

Names & Labels for jenkins_node_*_builds Metrics

Can't you just add another option, something like "jenkins-prefix" which defaults to "jenkins"

jenkins_node_my_node1_builds

i.e. "jenkins-prefix: myjenkins" results in: could become:

myjenkins_node_my_node1_builds

This would allow a single prometheus to scrape may jenkins and the metric sets would be distinguised by name, or, alternatively, add a label:

jenkins_node_my_node1_builds{namepace="namespace"}

Since namespace is already configurable.

james-m-tubbs

comment created time in 21 days

issue commentjenkinsci/prometheus-plugin

Prometheus plugins exception in log results in empty page when scraping

I can confirm that removing the Count * options, will output the metrics from Jenkins again.

mmedum

comment created time in 22 days

issue commentjenkinsci/prometheus-plugin

Prometheus plugins exception in log results in empty page when scraping

same behavior on windows 2012R2, Jenkins 2.249.2 LTS with plugin 2.0.8.

mmedum

comment created time in 22 days

issue closedjenkinsci/prometheus-plugin

Prometheus plugin 2.0.8 reports empty page

Steps to reproduce

How can we reproduce the issue

  1. upgrade to 2.0.8
  2. restart jenkins
  3. navigate to http://<dns of jenkins>/prometheus/

Expected behaviour

Normally prometheus data is presented as text

Actual behaviour

blank page, no content.

Server configuration

Operating system: Windows 2012R2

Jenkins Version: 2.249.2 LTS

Plugin Version: 2.0.8

Notify

@markyjackson-taulia

closed time in 22 days

advorsky73

issue commentjenkinsci/prometheus-plugin

Prometheus plugin 2.0.8 reports empty page

missed another report already stating this. sorry.

advorsky73

comment created time in 22 days

issue openedjenkinsci/prometheus-plugin

Prometheus plugin 2.0.8 reports empty page

Steps to reproduce

How can we reproduce the issue

  1. upgrade to 2.0.8
  2. restart jenkins
  3. navigate to http://<dns of jenkins>/prometheus/

Expected behaviour

Normally prometheus data is presented as text

Actual behaviour

blank page, no content.

Server configuration

Operating system: Windows 2012R2

Jenkins Version: 2.249.2 LTS

Plugin Version: 2.0.8

Notify

@markyjackson-taulia

created time in 22 days

issue commentjenkinsci/prometheus-plugin

Prometheus plugins exception in log results in empty page when scraping

Similar issue here. Since plugin 2.0.8 upgrade: There is a ClassCastException in the log and the /prometheus page is empty

SEVERE h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (prometheus_async_worker thread/3301139) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. java.lang.ClassCastException

Note: As soon as I disable the five "Count Duration *" settings of the plugin in "Manage Jenkins", the error log entry is gone and the metrics are back.

mmedum

comment created time in 22 days

issue commentjenkinsci/prometheus-plugin

Prometheus plugins exception in log results in empty page when scraping

See also: https://issues.jenkins-ci.org/browse/JENKINS-63998

mmedum

comment created time in 22 days

issue commentjenkinsci/prometheus-plugin

Prometheus plugins exception in log results in empty page when scraping

Many of the jobs running on my jenkins installation is permanent running jobs, where each run spawns one or more new runs, which will never be stopped. The problem seems to have appeared after Oct 25.

mmedum

comment created time in 23 days

issue openedjenkinsci/prometheus-plugin

Prometheus plugins exception in log results in empty page when scraping

Hello,

When navigating to the webpage/prometheus, I am receiving an empty page with a 200 status code. In my logs I see:

A thread (prometheus_async_worker thread/767) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.NullPointerException
	at org.jenkinsci.plugins.prometheus.JobCollector.lambda$appendJobMetrics$1(JobCollector.java:310)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at org.jenkinsci.plugins.prometheus.JobCollector.appendJobMetrics(JobCollector.java:310)
	at org.jenkinsci.plugins.prometheus.JobCollector.lambda$collect$0(JobCollector.java:220)
	at org.jenkinsci.plugins.prometheus.util.Jobs.forEachJob(Jobs.java:14)
	at org.jenkinsci.plugins.prometheus.JobCollector.collect(JobCollector.java:214)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144)
	at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
	at org.jenkinsci.plugins.prometheus.service.DefaultPrometheusMetrics.collectMetrics(DefaultPrometheusMetrics.java:51)
	at org.jenkinsci.plugins.prometheus.service.PrometheusAsyncWorker.execute(PrometheusAsyncWorker.java:41)
	at hudson.model.AsyncPeriodicWork.lambda$doRun$0(AsyncPeriodicWork.java:100)
	at java.lang.Thread.run(Thread.java:748)

I am also seeing

A thread (prometheus_async_worker thread/1054) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.lang.NullPointerException

In the prometheus logs, it seems to still receive information correctly from jobs, but I got no output when jenkins is scraped.

prometheus-plugin: 2.0.8 Jenkins: 2.235.3

created time in 23 days

push eventDavsX/dalsik

Dávid Kovács

commit sha 66b0ed6a831131db3f6937ed987297b74a02b65c

Remove empty line

view details

push time in a month

create barnchDavsX/dalsik

branch : feature/dual_layer_key_fix

created branch time in a month

issue commentjenkinsci/prometheus-plugin

Option to disable per-node metrics

I am not sure how you guys have your prometheus setup, but I drop these metrics at scrape time via something like this:

    - action: drop
      regex: ^jenkins_node_.+_builds$
      sourceLabels:
      - __name__
    - action: drop
      regex: ^jenkins_node_.+_builds_count$
      sourceLabels:
      - __name__
    - action: drop
      regex: ^label_jenkins_.*$
      sourceLabels:
      - __name__

I would recommend doing the same, until we get some traction on this feature / bug request.

languitar

comment created time in a month

pull request commentjenkinsci/prometheus-plugin

Add executor load statistics collector

Hey @markyjackson-taulia, I just resolved the merge conflicts.

Also, I was not looking for somebody else to write tests for the class that I've added, but rather pointed out that NO collector in this project is tested. Hence, I have no reference on how to implement tests for the collector I've added (ExecutorCollector) and was looking for your help ☝🏾 and you replied

I will give the examples for tests later tonight hopefully.

The issue I'm facing: I did not find any way to mock hudson.model.Label.loadStatistics as it is a public final transient field with the tooling you chose for your tests (in comparison, we heavily use Spock for our tests within my company).

This is how far I got:

package org.jenkinsci.plugins.prometheus;

import hudson.model.Label;
import hudson.model.LoadStatistics;
import io.prometheus.client.Collector;
import jenkins.model.Jenkins;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PowerMockIgnore;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
import static org.assertj.core.api.Assertions.assertThat;

import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

@RunWith(PowerMockRunner.class)
@PrepareForTest({Jenkins.class})
// PowerMockIgnore needed for: https://github.com/powermock/powermock/issues/864
@PowerMockIgnore({"com.sun.org.apache.xerces.*", "javax.xml.*", "org.xml.*", "org.w3c.*", "com.sun.org.apache.xalan.*"})
public class ExecutorCollectorTest {

    @Mock
    private Jenkins jenkins;

    @Mock
    private Label label1;

    @Mock
    private LoadStatistics loadStatistics1;

    @Mock
    private LoadStatistics.LoadStatisticsSnapshot loadStatisticsSnapshot1;

    @Mock
    private Set<Label> labels = new HashSet<Label>(){{
        add(label1);
    }};

    @Before
    public void setUp() {
        PowerMockito.mockStatic(Jenkins.class);
        PowerMockito.when(Jenkins.get()).thenReturn(jenkins);
        PowerMockito.when(jenkins.getLabels()).thenReturn(labels);

        PowerMockito.when(label1.getDisplayName()).thenReturn("Label 1");
        try {
            setFinalStatic(Label.class.getDeclaredField("loadStatistics"), loadStatistics1);
        }
        catch (SecurityException e) {Assert.fail("Exception: " + e);}
        catch (NoSuchFieldException e) {Assert.fail("Exception: " + e);}
        catch (Exception e) { Assert.fail("Exception: " + e); }

        PowerMockito.when(loadStatistics1.computeSnapshot()).thenReturn(loadStatisticsSnapshot1);
        PowerMockito.when(loadStatisticsSnapshot1.getAvailableExecutors()).thenReturn(1);
        PowerMockito.when(loadStatisticsSnapshot1.getBusyExecutors()).thenReturn(1);
        PowerMockito.when(loadStatisticsSnapshot1.getConnectingExecutors()).thenReturn(1);
        PowerMockito.when(loadStatisticsSnapshot1.getDefinedExecutors()).thenReturn(3);
        PowerMockito.when(loadStatisticsSnapshot1.getIdleExecutors()).thenReturn(2);
        PowerMockito.when(loadStatisticsSnapshot1.getOnlineExecutors()).thenReturn(1);
        PowerMockito.when(loadStatisticsSnapshot1.getQueueLength()).thenReturn(4);
    }

    @Test
    public void shouldCollectExecutorMetrics() {
        // given
        ExecutorCollector executorCollector = new ExecutorCollector();

        // when
        List<Collector.MetricFamilySamples> actual = executorCollector.collect();

        // then
        assertThat(actual.size()).isEqualTo(7);
    }

    static void setFinalStatic(Field field, Object newValue) throws Exception {
        // https://stackoverflow.com/a/23200308

        field.setAccessible(true);

        // remove final modifier from field
        Field modifiersField = Field.class.getDeclaredField("modifiers");
        modifiersField.setAccessible(true);
        modifiersField.setInt(field, field.getModifiers() & ~Modifier.FINAL);

        field.set(null, newValue);
    }
}

...but it throws java.lang.AssertionError: Exception: java.lang.NullPointerException.

msabastian

comment created time in a month

more