profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/kumarks1122/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Kumar kumarks1122 Bangalore I like programming...

cksachdev/code-red 2

code-red

kumarks1122/kssh 2

Shell Beautifier

kumarks1122/covid19india-react 1

📊 Source code of the main website

kumarks1122/Rails-with-an-ajax 1

How to we can use the ajax in rails

cksachdev/fuzzyjs 0

fuzzy search in one or more column values and sort based on most matched

EkStep-DevCon-2020/incubator-superset 0

Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application

kumarks1122/angular 0

One framework. Mobile & desktop.

kumarks1122/async 0

Async utilities for node and the browser

kumarks1122/athenaks 0

AthenaKS - Data Pipeline with Spark streaming

kumarks1122/central-heating-system-api 0

Central heating system

release rubocop/rubocop

v1.17.0

released time in 9 hours

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-24966: User Info exhaust to exclude rows for users who have not provided Consent

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100.png' alt='100.0%' width='16' height='16' /> 100.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' /> 0.0% Duplication

utk14

comment created time in 12 hours

pull request commentSunbird-Ed/sunbird-data-products

User Info exhaust to exclude rows for users who have not provided Consent

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100.png' alt='100.0%' width='16' height='16' /> 100.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' /> 0.0% Duplication

utk14

comment created time in a day

PR opened Sunbird-Ed/sunbird-data-products

User Info exhaust to exclude rows for users who have not provided Consent

https://project-sunbird.atlassian.net/browse/SB-24966

User Info exhaust to exclude rows for users who have not provided Consent

Type of change

Please choose appropriate options.

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes in the below checkboxes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • [ ] Ran Test A
  • [ ] Ran Test B

Test Configuration:

  • Software versions:
  • Hardware versions:

Checklist:

  • [ ] My code follows the style guidelines of this project
  • [ ] I have performed a self-review of my own code
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] New and existing unit tests pass locally with my changes
  • [ ] Any dependent changes have been merged and published in downstream modules
+88 -14

0 comment

3 changed files

pr created time in a day

create barnchSunbird-Ed/sunbird-data-products

branch : release-4.1.0

created branch time in 2 days

starteddonnemartin/system-design-primer

started time in 3 days

release apache/druid

druid-0.21.1

released time in 5 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #TG-955 feat: Handle throttling of data exhaust requests

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 6 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100.png' alt='100.0%' width='16' height='16' /> 100.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.2%' width='16' height='16' /> 2.2% Duplication

sowmya-dixit

comment created time in 5 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #TG-955 feat: Handle throttling of data exhaust requests

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 6 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100.png' alt='100.0%' width='16' height='16' /> 100.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.2%' width='16' height='16' /> 2.2% Duplication

sowmya-dixit

comment created time in 6 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #TG-955 feat: Handle throttling of data exhaust requests

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 6 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/90.png' alt='97.4%' width='16' height='16' /> 97.4% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='2.4%' width='16' height='16' /> 2.4% Duplication

sowmya-dixit

comment created time in 6 days

release rubocop/rubocop

v1.16.1

released time in 6 days

PR opened Sunbird-Ed/sunbird-data-products

Issue #TG-955 feat: Handle throttling of data exhaust requests

This PR includes below changes:

  • Limit requests processing by max batches per channel
  • Limit requests processing by max gb out per channel
  • Handling duplicate batch requests by just updating the record with already computed files

Type of change

Please choose appropriate options.

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes in the below checkboxes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • [ ] Ran Test A
  • [ ] Ran Test B

Test Configuration:

  • Software versions:
  • Hardware versions:

Checklist:

  • [ ] My code follows the style guidelines of this project
  • [ ] I have performed a self-review of my own code
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] New and existing unit tests pass locally with my changes
  • [ ] Any dependent changes have been merged and published in downstream modules
+250 -48

0 comment

6 changed files

pr created time in 6 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-24498: Added questionset objectType for contributions

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100.png' alt='100.0%' width='16' height='16' /> 100.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' /> 0.0% Duplication

ishawakankar

comment created time in 6 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-24498: Added questionset objectType for contributions

Kudos, SonarCloud Quality Gate passed!

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/100.png' alt='100.0%' width='16' height='16' /> 100.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' /> 0.0% Duplication

ishawakankar

comment created time in 6 days

created tagSunbird-Ed/sunbird-data-products

tagrelease-4.0.0_RC1

created time in 7 days

push eventSunbird-Ed/sunbird-data-products

Isha Wakankar

commit sha c3260c54dcf457ae393905018dd1cf5f80ebc3bd

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

view details

Isha Wakankar

commit sha 9c4565c8ae845f7c174c192820bea6cca2da3884

.

view details

Isha Wakankar

commit sha 932b612338975fe5da57e01d42d5bd904eecfa49

Added test cases

view details

Isha Wakankar

commit sha b2bb9f8d106f564f6f0867fee6d7a4a6d980b1bf

Changes for deleting old druid segments

view details

Isha Wakankar

commit sha 965217db281c0877b7284e37e4392feb5cd61451

Added loggers

view details

Isha Wakankar

commit sha 2847599130d96ff30309736355d6baa9e2977435

Review comment changes

view details

Isha Wakankar

commit sha 4650e49589ab599aa826a5f198dfc1bf24a032c2

Formatting code

view details

Isha Wakankar

commit sha d69bd266be82dfcb4f68fb471d7787bd4436992d

Refractoring code

view details

Isha Wakankar

commit sha c5e64c25fa14bb62ac12a132092bec0c72cc697c

Review comment changes

view details

Isha Wakankar

commit sha e11e32d4e87c382ad88d340cb16ac72583478af1

.

view details

Isha Wakankar

commit sha e937c35e461afa1f5dcb9b8165044d878124264d

Removed print statements

view details

Isha Wakankar

commit sha e7e607092e7c95c879eefc054efcfb1cef3c4d37

Test case changes

view details

Santhosh

commit sha 4c885686f551a452ee3f0b347430fd83cdd5a0e5

Merge pull request #399 from ishawakankar/SB-23540 Issue #SB-23540: Reports to be generated from Hawkeye infra changes

view details

push time in 7 days

PR merged Sunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

<!--do not remove this marker, its needed to replace info when ticket title is updated --> <!--jira-description-action-hidden-marker-start-->

<table> <td> <a href="https://project-sunbird.atlassian.net/browse/SB-23540" title="SB-23540" target="_blank"><img alt="Epic" src="https://project-sunbird.atlassian.net/images/icons/issuetypes/epic.svg" />SB-23540</a> Reports to be generated from Hawkeye infra using Druid </td></table> <br />

<!--jira-description-action-hidden-marker-end-->

+446 -19

2 comments

9 changed files

ishawakankar

pr closed time in 7 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

SonarCloud Quality Gate failed.

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/90.png' alt='95.0%' width='16' height='16' /> 95.0% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='5.7%' width='16' height='16' /> 5.7% Duplication

ishawakankar

comment created time in 7 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

SonarCloud Quality Gate failed.

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60.png' alt='88.3%' width='16' height='16' /> 88.3% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='5.7%' width='16' height='16' /> 5.7% Duplication

ishawakankar

comment created time in 7 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

SonarCloud Quality Gate failed.

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60.png' alt='88.5%' width='16' height='16' /> 88.5% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='5.7%' width='16' height='16' /> 5.7% Duplication

ishawakankar

comment created time in 7 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+package org.sunbird.analytics.sourcing++import java.util.Properties++import org.apache.spark.SparkContext+import org.sunbird.analytics.util.CourseUtils+import org.apache.spark.sql.functions.{when, _}+import org.apache.spark.sql._+import org.ekstep.analytics.framework.fetcher.DruidDataFetcher+import org.ekstep.analytics.framework.util.DatasetUtil.extensions+import org.ekstep.analytics.framework.{DruidQueryModel, FrameworkContext, IJob, JobConfig, Level, StorageConfig}+import org.ekstep.analytics.framework.util.{CommonUtil, JSONUtils, JobLogger, RestUtil}+import org.sunbird.analytics.exhaust.BaseReportsJob+import org.sunbird.cloud.storage.conf.AppConf++case class SourcingContents(primaryCategory: String, createdBy: String, count: Int)++object SourcingSummaryReport extends optional.Application with IJob with BaseReportsJob {++  implicit val className = "org.sunbird.analytics.sourcing.SourcingSummaryReport"+  val jobName: String = "SourcingSummaryReport"+  val db = AppConf.getConfig("postgres.db")+  val url = AppConf.getConfig("postgres.url") + s"$db"+  val connProperties = CommonUtil.getPostgresConnectionProps++  // $COVERAGE-OFF$ Disabling scoverage for main method+  def main(config: String)(implicit sc: Option[SparkContext], fc: Option[FrameworkContext]): Unit = {+    JobLogger.log(s"Started execution - $jobName", None, Level.INFO)+    implicit val jobConfig = JSONUtils.deserialize[JobConfig](config)+    implicit val frameworkContext: FrameworkContext = getReportingFrameworkContext()+    implicit val spark = openSparkSession(jobConfig)++    try {+      val res = CommonUtil.time(execute())+      JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1)))+    } finally {+      frameworkContext.closeContext()+      spark.close()+    }+  }++  // $COVERAGE-ON$ Enabling scoverage for all other functions+  def execute()(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val tables = JSONUtils.deserialize[Map[String, String]](JSONUtils.serialize(config.modelParams.get.getOrElse("tables", Map())))+    val programData = loadData(url, tables("programTable")).withColumnRenamed("status", "programStatus")+    val nominationData = loadData(url, tables("nominationTable"))+    val projectDf = programData.join(nominationData, Seq("program_id"), "outer")

tables should come from application conf. Only the host/ip specific configurations can come from job config

ishawakankar

comment created time in 7 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+package org.sunbird.analytics.sourcing++import java.util.Properties++import org.apache.spark.SparkContext+import org.sunbird.analytics.util.CourseUtils+import org.apache.spark.sql.functions.{when, _}+import org.apache.spark.sql._+import org.ekstep.analytics.framework.fetcher.DruidDataFetcher+import org.ekstep.analytics.framework.util.DatasetUtil.extensions+import org.ekstep.analytics.framework.{DruidQueryModel, FrameworkContext, IJob, JobConfig, Level, StorageConfig}+import org.ekstep.analytics.framework.util.{CommonUtil, JSONUtils, JobLogger, RestUtil}+import org.sunbird.analytics.exhaust.BaseReportsJob+import org.sunbird.cloud.storage.conf.AppConf++case class SourcingContents(primaryCategory: String, createdBy: String, count: Int)++object SourcingSummaryReport extends optional.Application with IJob with BaseReportsJob {++  implicit val className = "org.sunbird.analytics.sourcing.SourcingSummaryReport"+  val jobName: String = "SourcingSummaryReport"+  val db = AppConf.getConfig("postgres.db")+  val url = AppConf.getConfig("postgres.url") + s"$db"+  val connProperties = CommonUtil.getPostgresConnectionProps++  // $COVERAGE-OFF$ Disabling scoverage for main method+  def main(config: String)(implicit sc: Option[SparkContext], fc: Option[FrameworkContext]): Unit = {+    JobLogger.log(s"Started execution - $jobName", None, Level.INFO)+    implicit val jobConfig = JSONUtils.deserialize[JobConfig](config)+    implicit val frameworkContext: FrameworkContext = getReportingFrameworkContext()+    implicit val spark = openSparkSession(jobConfig)++    try {+      val res = CommonUtil.time(execute())+      JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1)))+    } finally {+      frameworkContext.closeContext()+      spark.close()+    }+  }++  // $COVERAGE-ON$ Enabling scoverage for all other functions+  def execute()(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val tables = JSONUtils.deserialize[Map[String, String]](JSONUtils.serialize(config.modelParams.get.getOrElse("tables", Map())))+    val programData = loadData(url, tables("programTable")).withColumnRenamed("status", "programStatus")+    val nominationData = loadData(url, tables("nominationTable"))+    val projectDf = programData.join(nominationData, Seq("program_id"), "outer")+      .select("program_id", "status", "rootorg_id", "user_id")++    process(projectDf)+  }++  def process(projectDf: DataFrame)(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val userDf = getUserDetails()+    val resultDf = userDf.join(projectDf, userDf.col("userId") === projectDf.col("user_id"), "outer")+      .withColumnRenamed("user_id", "contributorId")+      .withColumn("reportName", lit("SourcingSummaryReport"))+      .withColumn("timestamp", lit(System.currentTimeMillis()))+    JobLogger.log(s"resultDf count - ${resultDf.count()}", None, Level.INFO)++    val modelParams = config.modelParams.get+    val storageConfig = StorageConfig(AppConf.getConfig("cloud_storage_type"), "reports", "", Option(modelParams.getOrElse("storageKey", "druid_storage_account_key").toString), Option(modelParams.getOrElse("storageSecret", "druid_storage_account_secret").toString))+    resultDf.saveToBlobStore(storageConfig, "json", "sourcing",+      Option(Map("header" -> "true")), Option(List("reportName")))+    submitReportToDruid(modelParams)+  }++  def loadData(url: String, tableName: String)(implicit spark: SparkSession, config: JobConfig): DataFrame = {+    spark.read.jdbc(url, tableName, connProperties)+  }++  def submitReportToDruid(modelParams: Map[String, AnyRef]): Unit = {+    JobLogger.log(s"Submitting Druid Ingestion Task", None, Level.INFO)+    val segmentUrl = modelParams.getOrElse("druidSegmentUrl", "http://localhost:8081/druid/coordinator/v1/metadata/datasources/sourcing-model-snapshot/segments").asInstanceOf[String]+    val deleteSegmentUrl = modelParams.getOrElse("deleteSegmentUrl", "http://localhost:8081/druid/coordinator/v1/datasources/sourcing-model-snapshot/segments/").asInstanceOf[String]

Fetch only the host variable/baseurl from config. The API endpoint can be coded or can come from application.conf

ishawakankar

comment created time in 7 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+package org.sunbird.analytics.sourcing++import java.util.Properties++import org.apache.spark.SparkContext+import org.sunbird.analytics.util.CourseUtils+import org.apache.spark.sql.functions.{when, _}+import org.apache.spark.sql._+import org.ekstep.analytics.framework.fetcher.DruidDataFetcher+import org.ekstep.analytics.framework.util.DatasetUtil.extensions+import org.ekstep.analytics.framework.{DruidQueryModel, FrameworkContext, IJob, JobConfig, Level, StorageConfig}+import org.ekstep.analytics.framework.util.{CommonUtil, JSONUtils, JobLogger, RestUtil}+import org.sunbird.analytics.exhaust.BaseReportsJob+import org.sunbird.cloud.storage.conf.AppConf++case class SourcingContents(primaryCategory: String, createdBy: String, count: Int)++object SourcingSummaryReport extends optional.Application with IJob with BaseReportsJob {++  implicit val className = "org.sunbird.analytics.sourcing.SourcingSummaryReport"+  val jobName: String = "SourcingSummaryReport"+  val db = AppConf.getConfig("postgres.db")+  val url = AppConf.getConfig("postgres.url") + s"$db"+  val connProperties = CommonUtil.getPostgresConnectionProps++  // $COVERAGE-OFF$ Disabling scoverage for main method+  def main(config: String)(implicit sc: Option[SparkContext], fc: Option[FrameworkContext]): Unit = {+    JobLogger.log(s"Started execution - $jobName", None, Level.INFO)+    implicit val jobConfig = JSONUtils.deserialize[JobConfig](config)+    implicit val frameworkContext: FrameworkContext = getReportingFrameworkContext()+    implicit val spark = openSparkSession(jobConfig)++    try {+      val res = CommonUtil.time(execute())+      JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1)))+    } finally {+      frameworkContext.closeContext()+      spark.close()+    }+  }++  // $COVERAGE-ON$ Enabling scoverage for all other functions+  def execute()(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val tables = JSONUtils.deserialize[Map[String, String]](JSONUtils.serialize(config.modelParams.get.getOrElse("tables", Map())))+    val programData = loadData(url, tables("programTable")).withColumnRenamed("status", "programStatus")+    val nominationData = loadData(url, tables("nominationTable"))+    val projectDf = programData.join(nominationData, Seq("program_id"), "outer")+      .select("program_id", "status", "rootorg_id", "user_id")++    process(projectDf)+  }++  def process(projectDf: DataFrame)(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val userDf = getUserDetails()+    val resultDf = userDf.join(projectDf, userDf.col("userId") === projectDf.col("user_id"), "outer")+      .withColumnRenamed("user_id", "contributorId")+      .withColumn("reportName", lit("SourcingSummaryReport"))+      .withColumn("timestamp", lit(System.currentTimeMillis()))+    JobLogger.log(s"resultDf count - ${resultDf.count()}", None, Level.INFO)++    val modelParams = config.modelParams.get+    val storageConfig = StorageConfig(AppConf.getConfig("cloud_storage_type"), "reports", "", Option(modelParams.getOrElse("storageKey", "druid_storage_account_key").toString), Option(modelParams.getOrElse("storageSecret", "druid_storage_account_secret").toString))+    resultDf.saveToBlobStore(storageConfig, "json", "sourcing",+      Option(Map("header" -> "true")), Option(List("reportName")))+    submitReportToDruid(modelParams)+  }++  def loadData(url: String, tableName: String)(implicit spark: SparkSession, config: JobConfig): DataFrame = {+    spark.read.jdbc(url, tableName, connProperties)+  }++  def submitReportToDruid(modelParams: Map[String, AnyRef]): Unit = {+    JobLogger.log(s"Submitting Druid Ingestion Task", None, Level.INFO)+    val segmentUrl = modelParams.getOrElse("druidSegmentUrl", "http://localhost:8081/druid/coordinator/v1/metadata/datasources/sourcing-model-snapshot/segments").asInstanceOf[String]+    val deleteSegmentUrl = modelParams.getOrElse("deleteSegmentUrl", "http://localhost:8081/druid/coordinator/v1/datasources/sourcing-model-snapshot/segments/").asInstanceOf[String]+    val olderSegments = RestUtil.get[List[String]](segmentUrl)+    JobLogger.log(s"olderSegments - $olderSegments", None, Level.INFO)+    //disable older segments+    if (null != olderSegments) {+      olderSegments.foreach(segmentId => {+        val apiUrl = deleteSegmentUrl + segmentId+        RestUtil.delete(apiUrl)+        JobLogger.log(s"Deleted $segmentId", None, Level.INFO)+      })+    }++    val ingestionSpecPath: String = modelParams.getOrElse("specPath", "").asInstanceOf[String]+    val druidIngestionUrl: String = modelParams.getOrElse("druidIngestionUrl", "http://localhost:8081/druid/indexer/v1/task").asInstanceOf[String]+    CourseUtils.submitIngestionTask(druidIngestionUrl, ingestionSpecPath)+    JobLogger.log(s"Druid ingestion completed", None, Level.INFO)+  }++  def getUserDetails()(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig): DataFrame = {+    val openSaberDb = config.modelParams.get.getOrElse("dbName", "opensaberdb")+    val dbUrl = AppConf.getConfig("postgres.url") + openSaberDb++    val vUserData = loadData(dbUrl, "\"V_User\"")+      .select("userId")+    val vUserOrgData = loadData(dbUrl, "\"V_User_Org\"")+      .select("userId", "roles")

Fetch the table names from config

ishawakankar

comment created time in 7 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+package org.sunbird.analytics.sourcing++import java.util.Properties++import org.apache.spark.SparkContext+import org.sunbird.analytics.util.CourseUtils+import org.apache.spark.sql.functions.{when, _}+import org.apache.spark.sql._+import org.ekstep.analytics.framework.fetcher.DruidDataFetcher+import org.ekstep.analytics.framework.util.DatasetUtil.extensions+import org.ekstep.analytics.framework.{DruidQueryModel, FrameworkContext, IJob, JobConfig, Level, StorageConfig}+import org.ekstep.analytics.framework.util.{CommonUtil, JSONUtils, JobLogger, RestUtil}+import org.sunbird.analytics.exhaust.BaseReportsJob+import org.sunbird.cloud.storage.conf.AppConf++case class SourcingContents(primaryCategory: String, createdBy: String, count: Int)++object SourcingSummaryReport extends optional.Application with IJob with BaseReportsJob {++  implicit val className = "org.sunbird.analytics.sourcing.SourcingSummaryReport"+  val jobName: String = "SourcingSummaryReport"+  val db = AppConf.getConfig("postgres.db")+  val url = AppConf.getConfig("postgres.url") + s"$db"+  val connProperties = CommonUtil.getPostgresConnectionProps++  // $COVERAGE-OFF$ Disabling scoverage for main method+  def main(config: String)(implicit sc: Option[SparkContext], fc: Option[FrameworkContext]): Unit = {+    JobLogger.log(s"Started execution - $jobName", None, Level.INFO)+    implicit val jobConfig = JSONUtils.deserialize[JobConfig](config)+    implicit val frameworkContext: FrameworkContext = getReportingFrameworkContext()+    implicit val spark = openSparkSession(jobConfig)++    try {+      val res = CommonUtil.time(execute())+      JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1)))+    } finally {+      frameworkContext.closeContext()+      spark.close()+    }+  }++  // $COVERAGE-ON$ Enabling scoverage for all other functions+  def execute()(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val tables = JSONUtils.deserialize[Map[String, String]](JSONUtils.serialize(config.modelParams.get.getOrElse("tables", Map())))+    val programData = loadData(url, tables("programTable")).withColumnRenamed("status", "programStatus")+    val nominationData = loadData(url, tables("nominationTable"))+    val projectDf = programData.join(nominationData, Seq("program_id"), "outer")+      .select("program_id", "status", "rootorg_id", "user_id")++    process(projectDf)+  }++  def process(projectDf: DataFrame)(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val userDf = getUserDetails()+    val resultDf = userDf.join(projectDf, userDf.col("userId") === projectDf.col("user_id"), "outer")+      .withColumnRenamed("user_id", "contributorId")+      .withColumn("reportName", lit("SourcingSummaryReport"))+      .withColumn("timestamp", lit(System.currentTimeMillis()))+    JobLogger.log(s"resultDf count - ${resultDf.count()}", None, Level.INFO)++    val modelParams = config.modelParams.get+    val storageConfig = StorageConfig(AppConf.getConfig("cloud_storage_type"), "reports", "", Option(modelParams.getOrElse("storageKey", "druid_storage_account_key").toString), Option(modelParams.getOrElse("storageSecret", "druid_storage_account_secret").toString))+    resultDf.saveToBlobStore(storageConfig, "json", "sourcing",+      Option(Map("header" -> "true")), Option(List("reportName")))+    submitReportToDruid(modelParams)+  }++  def loadData(url: String, tableName: String)(implicit spark: SparkSession, config: JobConfig): DataFrame = {+    spark.read.jdbc(url, tableName, connProperties)+  }++  def submitReportToDruid(modelParams: Map[String, AnyRef]): Unit = {+    JobLogger.log(s"Submitting Druid Ingestion Task", None, Level.INFO)+    val segmentUrl = modelParams.getOrElse("druidSegmentUrl", "http://localhost:8081/druid/coordinator/v1/metadata/datasources/sourcing-model-snapshot/segments").asInstanceOf[String]+    val deleteSegmentUrl = modelParams.getOrElse("deleteSegmentUrl", "http://localhost:8081/druid/coordinator/v1/datasources/sourcing-model-snapshot/segments/").asInstanceOf[String]+    val olderSegments = RestUtil.get[List[String]](segmentUrl)+    JobLogger.log(s"olderSegments - $olderSegments", None, Level.INFO)+    //disable older segments+    if (null != olderSegments) {+      olderSegments.foreach(segmentId => {+        val apiUrl = deleteSegmentUrl + segmentId+        RestUtil.delete(apiUrl)+        JobLogger.log(s"Deleted $segmentId", None, Level.INFO)+      })+    }++    val ingestionSpecPath: String = modelParams.getOrElse("specPath", "").asInstanceOf[String]+    val druidIngestionUrl: String = modelParams.getOrElse("druidIngestionUrl", "http://localhost:8081/druid/indexer/v1/task").asInstanceOf[String]

Fetch only the host variable/baseurl from config. The API endpoint can be coded or can come from application.conf

ishawakankar

comment created time in 7 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

SonarCloud Quality Gate failed.

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/90.png' alt='94.9%' width='16' height='16' /> 94.9% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='5.7%' width='16' height='16' /> 5.7% Duplication

ishawakankar

comment created time in 8 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

SonarCloud Quality Gate failed.

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60.png' alt='88.1%' width='16' height='16' /> 88.1% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='5.7%' width='16' height='16' /> 5.7% Duplication

ishawakankar

comment created time in 8 days

pull request commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

SonarCloud Quality Gate failed.

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Bugs
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Vulnerabilities
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Security Hotspots
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' /> <img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' /> 0 Code Smells

<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60.png' alt='88.3%' width='16' height='16' /> 88.3% Coverage
<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10.png' alt='5.5%' width='16' height='16' /> 5.5% Duplication

ishawakankar

comment created time in 8 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+package org.sunbird.analytics.sourcing++import java.util.Properties++import org.apache.spark.SparkContext+import org.apache.spark.sql.Encoders+import org.sunbird.analytics.util.{CourseUtils, TextBookUtils}+import org.apache.spark.sql.functions.{when, _}+import org.apache.spark.sql._+import org.ekstep.analytics.framework.fetcher.DruidDataFetcher+import org.ekstep.analytics.framework.util.DatasetUtil.extensions+import org.ekstep.analytics.framework.{DruidQueryModel, FrameworkContext, IJob, JobConfig, Level, StorageConfig}+import org.ekstep.analytics.framework.util.{CommonUtil, JSONUtils, JobLogger, RestUtil}+import org.ekstep.analytics.model.{QueryDateRange, ReportConfig}+import org.sunbird.analytics.exhaust.BaseReportsJob+import org.sunbird.analytics.sourcing.FunnelReport.getStorageConfig+import org.sunbird.analytics.job.report.CollectionSummaryJobV2+import org.sunbird.cloud.storage.conf.AppConf++case class PostgresData(programId: String, programName: String)+case class SourcingContents(primaryCategory: String, createdBy: String, count: Int)++object SourcingSummaryReport extends optional.Application with IJob with BaseReportsJob {++  implicit val className = "org.sunbird.analytics.sourcing.SourcingSummaryReport"+  val jobName: String = "SourcingSummaryReport"+  val db = AppConf.getConfig("postgres.db")+  val url = AppConf.getConfig("postgres.url") + s"$db"+  val connProperties = CommonUtil.getPostgresConnectionProps+  val tables = Map("programTable" -> "program", "nominationTable" -> "nomination","vUser" -> "\"V_User\"", "vUserOrg" -> "\"V_User_Org\"")

The values are same across all envs, made changes to read from config

ishawakankar

comment created time in 8 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+package org.sunbird.analytics.sourcing++import java.util.Properties++import org.apache.spark.SparkContext+import org.apache.spark.sql.Encoders+import org.sunbird.analytics.util.{CourseUtils, TextBookUtils}+import org.apache.spark.sql.functions.{when, _}+import org.apache.spark.sql._+import org.ekstep.analytics.framework.fetcher.DruidDataFetcher+import org.ekstep.analytics.framework.util.DatasetUtil.extensions+import org.ekstep.analytics.framework.{DruidQueryModel, FrameworkContext, IJob, JobConfig, Level, StorageConfig}+import org.ekstep.analytics.framework.util.{CommonUtil, JSONUtils, JobLogger, RestUtil}+import org.ekstep.analytics.model.{QueryDateRange, ReportConfig}+import org.sunbird.analytics.exhaust.BaseReportsJob+import org.sunbird.analytics.sourcing.FunnelReport.getStorageConfig+import org.sunbird.analytics.job.report.CollectionSummaryJobV2+import org.sunbird.cloud.storage.conf.AppConf++case class PostgresData(programId: String, programName: String)+case class SourcingContents(primaryCategory: String, createdBy: String, count: Int)++object SourcingSummaryReport extends optional.Application with IJob with BaseReportsJob {++  implicit val className = "org.sunbird.analytics.sourcing.SourcingSummaryReport"+  val jobName: String = "SourcingSummaryReport"+  val db = AppConf.getConfig("postgres.db")+  val url = AppConf.getConfig("postgres.url") + s"$db"+  val connProperties = CommonUtil.getPostgresConnectionProps+  val tables = Map("programTable" -> "program", "nominationTable" -> "nomination","vUser" -> "\"V_User\"", "vUserOrg" -> "\"V_User_Org\"")++  // $COVERAGE-OFF$ Disabling scoverage for main method+  def main(config: String)(implicit sc: Option[SparkContext], fc: Option[FrameworkContext]): Unit = {+    JobLogger.log(s"Started execution - $jobName",None, Level.INFO)+    implicit val jobConfig = JSONUtils.deserialize[JobConfig](config)+    implicit val frameworkContext: FrameworkContext = getReportingFrameworkContext()+    implicit val spark = openSparkSession(jobConfig)++    try {+      val res = CommonUtil.time(execute())+      JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1)))+    } finally {+      frameworkContext.closeContext()+      spark.close()+    }+  }++  // $COVERAGE-ON$ Enabling scoverage for all other functions+  def execute()(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val programData = spark.read.jdbc(url, tables("programTable"), connProperties).withColumnRenamed("status","programStatus")+    val nominationData = spark.read.jdbc(url, tables("nominationTable"), connProperties)+    val projectDf = programData.join(nominationData, Seq("program_id"),"outer")+      .select("program_id","status","rootorg_id","user_id")+    JobLogger.log(s"Project count - ${projectDf.count()}",None, Level.INFO)++    process(projectDf)+  }++  def process(projectDf: DataFrame)(implicit spark: SparkSession, fc: FrameworkContext, config: JobConfig) = {+    val userDf = getUserDetails()+    JobLogger.log(s"User data count - ${userDf.count()}",None, Level.INFO)+    val resultDf = userDf.join(projectDf, userDf.col("userId") === projectDf.col("user_id"), "outer")+      .withColumnRenamed("user_id","contributorId")+      .withColumn("reportName", lit("SourcingReport"))+      .withColumn("timestamp", lit(System.currentTimeMillis()))+    JobLogger.log(s"resultDf count - ${resultDf.count()}",None, Level.INFO)++    val modelParams = config.modelParams.get+    val storageConfig = StorageConfig(AppConf.getConfig("cloud_storage_type"), "reports", "", Option(modelParams.getOrElse("storageKey","druid_storage_account_key").toString), Option(modelParams.getOrElse("storageSecret","druid_storage_account_secret").toString))+    resultDf.saveToBlobStore(storageConfig, "json", "sourcing",

sourcing is the folder name where all the other reports for vdn are being stored. Have tested the changes, they are working as expected

ishawakankar

comment created time in 8 days

Pull request review commentSunbird-Ed/sunbird-data-products

Issue #SB-23540: Reports to be generated from Hawkeye infra changes

+{+  "type": "index",+  "spec": {+    "dataSchema": {+      "dataSource": "sourcing-model-snapshot",+      "parser": {+        "type": "string",+        "parseSpec": {+          "format": "json",+          "flattenSpec": {+            "useFieldDiscovery": false,+            "fields": [{+              "type": "root",+              "name": "program_id",+              "expr": "program_id"+            },+              {+                "type": "root",+                "name": "status",+                "expr": "status"+              },+              {+                "type": "root",+                "name": "rootorg_id",+                "expr": "rootorg_id"+              },+              {+                "type": "root",+                "name": "userId",+                "expr": "userId"+              },+              {+                "type": "root",+                "name": "userType",+                "expr": "userType"+              },+              {+                "type": "root",+                "name": "contributorId",+                "expr": "contributorId"+              },+              {+                "type": "root",+                "name": "totalContributedContent",+                "expr": "totalContributedContent"+              },+              {+                "type": "root",+                "name": "primaryCategory",+                "expr": "primaryCategory"+              },+              {+                "type": "root",+                "name": "createdBy",+                "expr": "createdBy"+              }+            ]+          },+          "dimensionsSpec": {+            "dimensions": [{+              "type": "string",+              "name": "program_id"+            },+              {+                "type": "string",+                "name": "status"+              },+              {+                "type": "string",+                "name": "rootorg_id"+              },+              {+                "type": "string",+                "name": "userId"+              },+              {+                "type": "string",+                "name": "userType"+              },+              {+                "type": "string",+                "name": "contributorId"+              },+              {+                "type": "string",+                "name": "primaryCategory"+              },+              {+                "type": "string",+                "name": "createdBy"+              }+            ],+            "dimensionsExclusions": []+          },+          "timestampSpec": {+            "column": "timestamp",+            "format": "auto"+          }+        }+      },+      "metricsSpec": [+        {+          "type":"longSum",+          "name": "totalContributedContent",+          "fieldName":"totalContributedContent"+        }+      ],+      "granularitySpec": {+        "type": "uniform",+        "segmentGranularity": "day",+        "queryGranularity": "none",+        "rollup": true+      }+    },+    "ioConfig": {+      "type": "index",+      "firehose": {+        "type": "static-azure-blobstore",+        "blobs": [{+          "container": "reports",+          "path": "/sourcing/SourcingReports.json"

Updated report name

ishawakankar

comment created time in 8 days