profile
viewpoint
Simeon Simeonov ssimeonov Swoop, Inc. Boston, MA http://blog.simeonov.com ["co-found", "hack", "create", "enjoy", "play", "invest", "partner"].sort(function(){return 0.5-Math.random()})

misshie/interval-tree 34

A Ruby implementation of augmented interval tree

ssimeonov/couponsgem 9

Quick and dirty coupon code management

ssimeonov/composable_state_machine 1

Tiny state machine implementation with clean separation between transitions, transition logic & state management.

ssimeonov/tilt 1

Generic interface to multiple Ruby template engines

ssimeonov/amphtml 0

AMP HTML source code, samples, and documentation. See below for more info.

ssimeonov/bandit 0

A multi-armed bandit optimization framework for Rails

ssimeonov/bandit-1 0

R package for split test/one-armed bandit analysis

ssimeonov/Bandits 0

A JavaScript demo of some multi-armed bandits algorithms

ssimeonov/BanditsBook 0

Code for my book on Multi-Armed Bandit Algorithms

ssimeonov/bestcase 0

Bestcase is an A/B and multivariate testing library for Clojure.

issue commentswoop-inc/spark-records

Fix the GitHub page for this project

That is very odd: the gh-pages branch hasn't been updated in years.

MrPowers

comment created time in 8 days

push eventswoop-inc/spark-records

MrPowers

commit sha 412fe5125b61b00b490e01f249ae4af1ce8b714b

Remove spark-test-sugar dependency

view details

MrPowers

commit sha 57aa9b727662df0f776e62cbb757f5efd01f7830

Use 2 cores when running tests

view details

Simeon Simeonov

commit sha 5ea2935e015186c284e2ed87494d744fc3e182d3

Merge pull request #9 from MrPowers/remove-test-sugar Remove spark-test-sugar dependency

view details

push time in 11 days

PR merged swoop-inc/spark-records

Remove spark-test-sugar dependency

Removing the spark-test-sugar dependency will let us cross compile this library with Scala 2.12.

+33 -12

0 comment

5 changed files

MrPowers

pr closed time in 11 days

PullRequestReviewEvent

Pull request review commentswoop-inc/spark-records

Remove spark-test-sugar dependency

 package examples.fancy_numbers +import com.swoop.spark.SparkSessionTestWrapper import com.swoop.spark.records._-import com.swoop.spark.test.SparkSqlSpec import org.apache.spark.sql.Dataset import org.apache.spark.storage.StorageLevel  -class SparkTest extends ExampleSpec with SparkSqlSpec with TestNegative5To100 {+class SparkTest extends ExampleSpec with SparkSessionTestWrapper with TestNegative5To100 { +  val sc = spark.sparkContext   lazy val dc = SimpleDriverContext(sc)   lazy val jc = dc.jobContext(SimpleJobContext)   lazy val ds = recordsDataset(-5 to 100, jc)   lazy val records = ds.collect    "in an integration test" - {     implicit val env = FlatRecordEnvironment()-    val sqlContext = sqlc-    import sqlContext.implicits._+    import spark.implicits._      behave like fancyRecordBuilder(records, jc)      "should build records with Spark" in {       ds.count should be(105)     }+

No need for the extra whitespace for such simple tests: the description and { ... } create sufficient visual separation without wasting vertical space.

MrPowers

comment created time in 11 days

Pull request review commentswoop-inc/spark-records

Remove spark-test-sugar dependency

+package com.swoop.spark++import org.apache.spark.sql.SparkSession++trait SparkSessionTestWrapper {++  lazy val spark: SparkSession = {+    SparkSession+      .builder()+      .master("local")

Running tests with a single worker and a single partition can hide errors related to parallel execution. We tend to run local tests with 2 workers.

MrPowers

comment created time in 11 days

Pull request review commentswoop-inc/spark-records

Remove spark-test-sugar dependency

+package com.swoop.spark++import org.apache.spark.sql.SparkSession++trait SparkSessionTestWrapper {++  lazy val spark: SparkSession = {+    SparkSession+      .builder()+      .master("local")+      .appName("spark-records")+      .config(

What's the value of one config setting being split across 4 lines?

MrPowers

comment created time in 11 days

Pull request review commentswoop-inc/spark-records

Remove spark-test-sugar dependency

+# Set everything to be logged to the console+log4j.rootCategory=ERROR, console+log4j.appender.console=org.apache.log4j.ConsoleAppender+log4j.appender.console.target=System.err+log4j.appender.console.layout=org.apache.log4j.PatternLayout+log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n++# Settings to quiet third party logs that are too verbose+log4j.logger.org.eclipse.jetty=WARN+log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR+log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=WARN+log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=WARN

Missing newline.

MrPowers

comment created time in 11 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventswoop-inc/spark-records

MrPowers

commit sha 5b65aaeec23a613fccc74cabc58d1f831af91ed9

Add resolver to fetch the spark-test-sugar dependency

view details

Simeon Simeonov

commit sha b9e65b900b8b0af26d0717d0fbc1c689c9948123

Merge pull request #7 from MrPowers/second-attempt-fix-build Add resolver to fetch the spark-test-sugar dependency

view details

push time in 20 days

PR merged swoop-inc/spark-records

Add resolver to fetch the spark-test-sugar dependency

Thanks for building this library 😄

I was getting this error when running sbt test:

[info] Resolving com.swoop#spark-test-sugar_2.11;1.5.0 ...
[warn] 	module not found: com.swoop#spark-test-sugar_2.11;1.5.0
[warn] ==== local: tried
[warn]   /Users/matthewpowers/.ivy2/local/com.swoop/spark-test-sugar_2.11/1.5.0/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/com/swoop/spark-test-sugar_2.11/1.5.0/spark-test-sugar_2.11-1.5.0.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /Users/matthewpowers/.sbt/preloaded/com.swoop/spark-test-sugar_2.11/1.5.0/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////Users/matthewpowers/.sbt/preloaded/com/swoop/spark-test-sugar_2.11/1.5.0/spark-test-sugar_2.11-1.5.0.pom
[warn] ==== tpolecat: tried
[warn]   http://dl.bintray.com/tpolecat/maven/com/swoop/spark-test-sugar_2.11/1.5.0/spark-test-sugar_2.11-1.5.0.pom
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.swoop#spark-test-sugar_2.11;1.5.0: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::

It was looking in http://dl.bintray.com/tpolecat/maven/com/swoop/spark-test-sugar_2.11 for spark-test-sugar instead of in https://dl.bintray.com/swoop-inc/maven/. Adding the resolver in the build.sbt file fixes this build on my machine. Quite the strange error message!

+2 -0

0 comment

2 changed files

MrPowers

pr closed time in 20 days

PullRequestReviewEvent

startedtonsky/datascript

started time in 2 months

startedmikub/titanoboa

started time in 2 months

startedhasura/graphql-engine

started time in 2 months

fork ssimeonov/db-readings

Readings in Databases

fork in 2 months

startedrxin/db-readings

started time in 2 months

push eventswoop-inc/fast_cache

Simeon Simeonov

commit sha 5111a30f829604c5ddab5e7cc6563785ae6df5a8

Upgrades Ruby to 2.7.1

view details

push time in 2 months

create barnchswoop-inc/fast_cache

branch : ss_ruby271

created branch time in 2 months

push eventswoop-inc/composable_state_machine

Simeon Simeonov

commit sha b08271971806e591ceda9931e63b14e265257898

Updates Ruby to 2.7.1

view details

push time in 2 months

startedswoop-inc/cuttle

started time in 2 months

push eventswoop-inc/fast_cache

Simeon Simeonov

commit sha 9245e3afcd787fa392d18dabf526b41c531cd72f

Removes Travis CI as it is being replaced

view details

push time in 3 months

push eventswoop-inc/fast_cache

Simeon Simeonov

commit sha 266a2d8ab80750d94021537841477a9a8cbe0404

Removes broken link

view details

push time in 3 months

push eventswoop-inc/composable_state_machine

Simeon Simeonov

commit sha ca3fd7c843843c88a529b92d4147cd20b448a95b

Removes broken link

view details

push time in 3 months

push eventswoop-inc/composable_state_machine

Simeon Simeonov

commit sha 89da08d2ec2069bd6ed67496f2354b9244b8b6f6

Removes Gemnasium dependency

view details

push time in 3 months

push eventswoop-inc/fast_cache

Simeon Simeonov

commit sha 98cc98b591ba4c9b9843817b7218af2a94584fb6

Removed Gemnasium dependency

view details

push time in 3 months

more