profile
viewpoint

charleswhchan/dwave-ocean-qubo-examples 2

My attempt to implement problems described "A Tutorial on Formulating and Using QUBO Models" (https://arxiv.org/pdf/1811.11538.pdf) using D-Wave Ocean SDK in Python and solving on D-Wave Quantum Computer

charleswhchan/celery 1

Distributed Task Queue (development branch)

charleswhchan/amazon-kinesis-client-python 0

Amazon Kinesis Client Library for Python

charleswhchan/ansible 0

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy. Avoid writing scripts or custom code to deploy and update your applications— automate in a language that approaches plain English, using SSH, with no agents to install on remote systems.

charleswhchan/ansible-lint 0

Best practices checker for Ansible

charleswhchan/apidocjs.com 0

apidocjs.com Website

charleswhchan/atom 0

The hackable text editor

charleswhchan/awesome-go 0

A curated list of awesome Go frameworks, libraries and software

charleswhchan/aws-cli 0

Universal Command Line Interface for Amazon Web Services

charleswhchan/backbone 0

Give your JS App some Backbone with Models, Views, Collections, and Events

startedmlarocca/AlgorithmsAndDataStructuresInAction

started time in 20 days

starteddjango-guardian/django-guardian

started time in a month

starteddfunckt/django-rules

started time in a month

startedvintasoftware/django-role-permissions

started time in a month

startedvintasoftware/django-role-permissions

started time in a month

startedjazzband/django-authority

started time in a month

startedyandex/zero-downtime-migrations

started time in a month

startedReact-Proto/react-proto

started time in a month

startedocadotechnology/django-closuretree

started time in 2 months

starteddjango-mptt/django-mptt

started time in 2 months

starteddjango-treebeard/django-treebeard

started time in 2 months

issue commentlocalstack/localstack

Simulate S3 BadDigest and InvalidDigest errors

Thanks @whummer, I can confirm with the latest image, the not base64 encoded checksum case is now fixed (and returns InvalidDigest). 👏👍

I did mention another case in my last post (not in gist): 12345678901234567890 returns BadDigest while 1234567890123456789 returns InvalidDigest. It don't really matter to me but I am not sure if you want to take a look also?

charleswhchan

comment created time in 2 months

issue commentlocalstack/localstack

Simulate S3 BadDigest and InvalidDigest errors

Hi @whummer : I really appreciate your effort in troubleshooting this issue 👍

After reviewing your tests and comparing to my test, I found something interesting that triggers the error. I have created a gist

Specifically, if you refer to the checksum used in Test #3:

	fmt.Println("Test #3: InvalidDigest ---------------------------------------")
	contentMD5 = "not base64 encoded checksum"
	uploadPartInput = &s3.UploadPartInput{
		Body:       aws.ReadSeekCloser(bytes.NewReader(data)),
		Bucket:     aws.String(bucket),
		Key:        aws.String(key),
		ContentMD5: aws.String(contentMD5),
		PartNumber: aws.Int64(int64(partNumber)),
		UploadId:   aws.String(uploadID),
	}

	fmt.Println("UploadPart ...")
	_, err = svc.UploadPart(uploadPartInput)
	fmt.Printf("expect InvalidDigest; actual %+v\n", err)

Output

Test #3: InvalidDigest ---------------------------------------
UploadPart ...
expect InvalidDigest; actual BadDigest: The Content-MD5 you specified did not match what we received.

I did a bit more investigation and it appears it might have to something with the length of the contentMD5? If the length is <= 19, localstack would return InvalidDigest (correctly): contentMD5 = "1234567890123456789" Otherwise, localstack would return BadDigest? contentMD5 = "12345678901234567890"

Strange problem. I hope this information helps 😃

charleswhchan

comment created time in 2 months

issue commentRichardKnop/machinery

Redis SSL

@yap-studios : sorry but I don't have permission. Please check with @RichardKnop. 😃

yap-studios

comment created time in 2 months

issue commentRichardKnop/machinery

Redis SSL

Yeah. PR looks reasonable to me.

yap-studios

comment created time in 2 months

issue commentlocalstack/localstack

Incorrect response code when Downloading S3 object with invalid range

Thanks @whummer for the quick turnaround. You are right, it is working with the following image:

$ docker pull localstack/localstack:latest
latest: Pulling from localstack/localstack
8a4e9a21a873: Pull complete 
Digest: sha256:3db28dbf80b9587376d11306398e1d2e8ee32bfbb5bf7528d4dd84a0f314cdc5
Status: Downloaded newer image for localstack/localstack:latest
docker.io/localstack/localstack:latest

Please close the ticket. 👍

charleswhchan

comment created time in 2 months

issue commentlocalstack/localstack

Incorrect response code when Downloading S3 object with invalid range

Hi @whummer, I tried the latest docker image and I am still seeing the error: "error":"RequestedRangeNotSatisfiable: Requested Range Not Satisfiable\n\tstatus code: 416

Looking at the test case and comparing to my code. I am using Download().

Perhaps the fix needs to be applied some where else as well? Maybe a test using download_file() or download_fileobj() will help expose the problem?

charleswhchan

comment created time in 2 months

issue commentlocalstack/localstack

Simulate S3 BadDigest and InvalidDigest errors

Hi @whummer.

I just tried the latest docker image (link) and it seems to broken in a different way -- it is now returning BadDigest all the time.

I am using go-lang, and here is my 2 test cases:

Test 1. checksum := "not base64 encoded checksum"

  • Expect InvalidDigest
  • Actual 0.11.1: InvalidDigest
  • Actual latest: BadDigest

Test 2

		checksum := base64.StdEncoding.EncodeToString(wrongCheckSum[:])```
- Expect `BadDigest`
- Actual 0.11.1: `InvalidDigest` ❌  
- Actual `latest`: `BadDigest` ✅ 

Note: I also run the same test against S3 and MinIO to to confirm the expected behaviour.
charleswhchan

comment created time in 2 months

issue commentRichardKnop/machinery

Redis SSL

The factory currently looks for the prefix "redis://": https://github.com/RichardKnop/machinery/blob/master/v1/factories.go#L41

Looking at the underlying Redis libraries, both of them supports rediss://:

  • redigo https://github.com/gomodule/redigo/pull/208
  • go-redis https://github.com/go-redis/redis/pull/420

It should be pretty easy to modify the factory to support this. Want to give this a try and create a PR for it?

yap-studios

comment created time in 2 months

startedhshoff/vx

started time in 2 months

startedplouc/nivo

started time in 2 months

startedFormidableLabs/victory

started time in 2 months

starteduber/react-vis

started time in 2 months

startedgithub/super-linter

started time in 2 months

issue openedlocalstack/localstack

Incorrect response code when Downloading S3 object with invalid range

Type of request: This is a ...

[x] bug report [ ] feature request

Detailed description

When download invalid a range of bytes from S3 with an invalid range. localstack returns an incorrect error code RequestedRangeNotSatisfiable instead of InvalidRange

Expected behavior

(from S3) Err: InvalidRange: The requested range is not satisfiable

Actual behavior

(from localstack) RequestedRangeNotSatisfiable

Client code (AWS SDK code snippet, or sequence of "awslocal" commands)

	// Test S3 error code when invalid byte range is specified
	// Note: the file should be smaller than the range specified
	contentLength := 1000000
	rangeByteHeader := "bytes=1000-1000000"
        fmt.Printf("    range:  %v\n", rangeByteHeader)

	getObjectInput := &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
		Range:  aws.String(rangeByteHeader),
	}
	buffer := aws.NewWriteAtBuffer(make([]byte, contentLength))

	fmt.Println("downloader.Download ...")
	size, err := downloader.Download(buffer, getObjectInput)
	_ = size
	fmt.Printf("Err: %+v\n", err)

created time in 2 months

issue commenttonybaloney/pycharm-security

Update insecure.json regularly

Good to know. Thanks @tonybaloney

The URL option works for me. I am happy with this solution so feel free to close the ticket if appropriate.

charleswhchan

comment created time in 2 months

issue commentlocalstack/localstack

Simulate S3 BadDigest and InvalidDigest errors

Hi @whummer, thanks for looking into this.

What I am trying to do is write a test to write to S3 with localstack and produce conditions that would lead to BadDigest and InvalidDigest errors. This is so that I can check the error handling is working correctly in my code.

Based on my understanding of the AWS REST API. In the code you listed above, when "actual != expected", localstack should return BadDigest as the checksum does not match. Whereas if the content of "Context-MD5" is not a base64 encoded MD5, then the API should return InvalidDigest.

Does this help?

charleswhchan

comment created time in 2 months

issue openedtonybaloney/pycharm-security

Update insecure.json regularly

It looks like safety-db updates their database once a month. https://github.com/pyupio/safety-db/commits/master/data/insecure.json

Is it possible for this project to follow the same cadence? There were some false warnings that was fixed after Apr 14th release.

created time in 2 months

issue openedlocalstack/localstack

Simulate S3 BadDigest and InvalidDigest errors

Type of request: This is a ...

[ ] bug report [x] feature request

Detailed description

As a Developer, I want to be able to simulate BadDigest and InvalidDigest errors, So that I can test the error handling logic

Expected behavior

https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html BadDigest: The Content-MD5 you specified did not match what we received. InvalidDigest: The Content-MD5 you specified is not valid.

Actual behavior

localstack returns InvalidDigest when MD5 does not match payload.

created time in 2 months

delete branch charleswhchan/machinery

delete branch : fix-typo-in-config.go

delete time in 2 months

delete branch charleswhchan/machinery

delete branch : typo-task-status

delete time in 2 months

push eventcharleswhchan/machinery

Charles Chan

commit sha 3450bb4c17bc38569a9909fc9818f224446a4075

Fix typo in config.goRe-apply 342ef9fe

view details

push time in 2 months

PR opened RichardKnop/machinery

Fix typo in config.go

Re-apply 342ef9fe

+1 -1

0 comment

1 changed file

pr created time in 2 months

create barnchcharleswhchan/machinery

branch : fix-typo-in-config.go

created branch time in 2 months

issue commentRichardKnop/machinery

[Question] Calling Task without server

I see what you mean. Interesting concept. Unfortunately it is not something I had experience with.

Comparing the JSON format in the README against tasks.Signature, I can see newer changes in the struct. I suspect the README might not be the most up to date.

You might want to have a look at https://github.com/RichardKnop/machinery/blob/master/v1/brokers/amqp/amqp.go#L186. I can see it call json.Marshal(signature), so you could write a small golang server to publish a small task to print out the exact message + format being published to the queue.

tarunwadhwa13

comment created time in 2 months

create barnchcharleswhchan/machinery

branch : typo-task-status

created branch time in 2 months

issue commentRichardKnop/machinery

Question: I need to get timestamp for task completion time.

The easiest solution is to log a message in your task, or use TaskResult

siredwin

comment created time in 2 months

issue commentRichardKnop/machinery

Parameter passing between chain tasks does not match the document

Docs updated. See #339.

WisperDin

comment created time in 2 months

issue commentRichardKnop/machinery

[Question] Calling Task without server

If I understand your question: you want to be able to test your task without invoking queue/worker?

Since tasks are just defined as regular functions, they call be called directly as well to test/debug the implementation. This might help you isolate issues with Registration, Signatures, or Sending Tasks.

Using Add task as an example:

func Add(args ...int64) (int64, error) {
  sum := int64(0)
  for _, arg := range args {
    sum += arg
  }
  return sum, nil
}

You can call it directly:

result, err := Add(1, 2)
tarunwadhwa13

comment created time in 2 months

pull request commentdwavesystems/dwave-cloud-client

Add support for client-side certificate auth

Thanks! LGTM. You can leave this as a branch and we will test and leave a comment.

No rush to merge this PR until we know it's working e2e.

randomir

comment created time in 2 months

issue commentcelery/django-celery-beat

Multiple instances

You can add a lock to prevent multiple instances of beat from running in parallel.

Also see https://blog.heroku.com/redbeat-celery-beat-scheduler:

Finally, we added a simple lock that prevents multiple Beat daemons from running concurrently. This can sometimes be a problem for Heroku customers when they scale up from a single worker or during development.

eikebartels

comment created time in 2 months

issue commentRichardKnop/machinery

The log output should be customizable

@MerlinFeng : Did you see how to specify a custom logger in the README? https://github.com/RichardKnop/machinery#custom-logger

MerlinFeng

comment created time in 2 months

pull request commentRichardKnop/machinery

Fix 413 - (Redis) Simplify TTL

@RichardKnop : can you please close #413 as well? Thanks.

charleswhchan

comment created time in 2 months

delete branch charleswhchan/machinery

delete branch : fix-413

delete time in 2 months

delete branch charleswhchan/machinery

delete branch : fix-399

delete time in 2 months

PR opened RichardKnop/machinery

Fix 413 - (Redis) Simplify TTL

Set TTL for the key at the same time when setting the value

+26 -33

0 comment

2 changed files

pr created time in 2 months

create barnchcharleswhchan/machinery

branch : fix-413

created branch time in 2 months

PR opened RichardKnop/machinery

Fix #339 - Document argument passing behavior with chains

Clarify example in README

+6 -3

0 comment

1 changed file

pr created time in 2 months

push eventcharleswhchan/machinery

Charles Chan

commit sha be459e8bb3548e6f2ad9a006b91a8ccac9399a12

Fix #339 - Document argument passing behavior with chains

view details

push time in 2 months

push eventcharleswhchan/machinery

Charles Chan

commit sha a45fabae999215921518b8141ecbf503c40f929f

Fix #339 - Document argument passing behavior with chains

view details

push time in 2 months

create barnchcharleswhchan/machinery

branch : fix-399

created branch time in 2 months

push eventcharleswhchan/machinery

Julian Wachholz

commit sha d56568990fd1f6a98f6edb85ad30196bbd800ba0

Add AMQP queue declaration configuration options

view details

Julian Wachholz

commit sha aaf643692c8ed30e09702c8118a27aed91debc4f

Add priority field to tasks.Signature

view details

Patrik Cyvoct

commit sha 956b8d06941f0753c4fef86466f6e9351e719686

Add multiple amqp brokers Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>

view details

Patrik Cyvoct

commit sha dc01f0e93c30b5529277648b1de7e35dda89283f

Add test for multiple amqp brokers Signed-off-by: Patrik Cyvoct <patrik@ptrk.io>

view details

mjarkk

commit sha e825abf93d526c7505ba3ec4e9cf8d8ebb6ca693

Added IgnoreWhenTaskNotRegistered option

view details

Wing Gao

commit sha b5602ccb7660e797bea8f06713f9c6197d32be10

feat: use go-redis

view details

Wing Gao

commit sha c4bfb83d3eb21ca1db5f0e95bc93548ceb994474

feat(redis): add config DelayedTasksKey

view details

Wing Gao

commit sha 2e61561703453a7f360d4bf475f2c6b238a67a20

feat(redis): add backend

view details

Wing Gao

commit sha e946a7371676719453d2f5091258f67287a664f0

feat(redis): add test for cluster

view details

Uzair Khan

commit sha 422c46e392c7a1179b25ebb44f110dd35a514c3a

add option to stop task deletion useful when we want to sqs to send messages to dlq on repeated consumption and processing failure of erroneous messages

view details

Charles Chan

commit sha 35ad717e24d2b90082e3e6eae5c92c73366539ec

Update Retry Tasks description

view details

Richard Knop

commit sha edf30ef54891777f86a10dc4a098ff4c3dbc5e06

Merge pull request #459 from WingGao/go-redis support redis cluster

view details

Richard Knop

commit sha 60cb100280f2d4004ab6cc4d4ee7fc3952a9d0a1

Merge branch 'master' into tread-no-handeler-as-failed-task

view details

Richard Knop

commit sha 3b244ca307d79354671ce09b35d0594693a5d3dc

fix: fixed a bug in redis backend factory

view details

Richard Knop

commit sha 2bcf3d9600540022e5a61de732951620e7b9aa12

Merge pull request #456 from mjarkk/tread-no-handeler-as-failed-task Proposal to add IgnoreWhenTaskNotRegistered option

view details

Richard Knop

commit sha 41b91b22e49439e8ca193b3247f39297237f6884

Merge branch 'master' into add-option-to-stop-task-deletion

view details

Richard Knop

commit sha 61b341685baf8c6566b2ab6e1681fbe3cb280c89

Merge pull request #472 from Uzair614/add-option-to-stop-task-deletion add option to stop task deletion

view details

Richard Knop

commit sha 985720f53d8d13548659095e57876d34f86f2749

Merge branch 'master' into multiple_aqmp

view details

Richard Knop

commit sha ee2b5403862d8b1f14b4de38f197b511a1f44640

Merge pull request #470 from charleswhchan/patch-1 Update Retry Tasks description

view details

Richard Knop

commit sha 0d7c6f56e875a6b2e995826aaccb2ac8c8f86bd6

Merge pull request #428 from julianwachholz/feature/queuedeclareargs Add AMQP queue declaration configuration options

view details

push time in 2 months

issue commentRichardKnop/machinery

Celery compatibility

No - Machinery is NOT compatible with Celery.

sashker

comment created time in 2 months

issue commentRichardKnop/machinery

Secret Information is logged when new worker is launched.

As the PR had been merged. Perhaps this ticket can be closed?

yolossn

comment created time in 2 months

issue commentRichardKnop/machinery

release v1.7.5 with better dynamodb support

Hi @roopakv, I see v1.7.5 had been released on Apr 20: https://github.com/RichardKnop/machinery/releases/tag/v1.7.5

Could you close this issue? :smiley:

roopakv

comment created time in 2 months

delete branch charleswhchan/machinery

delete branch : issue-529-redis-default-concurrency

delete time in 3 months

delete branch charleswhchan/machinery

delete branch : issue-548

delete time in 3 months

create barnchcharleswhchan/machinery

branch : issue-548

created branch time in 3 months

issue commentRichardKnop/machinery

Is it support periodic task?

Related to #521

lannyMa

comment created time in 3 months

issue commentRichardKnop/machinery

Support for periodic task

Also consider https://github.com/jasonlvhit/gocron for scheduling periodic tasks.

CatchZeng

comment created time in 3 months

PR opened RichardKnop/machinery

Fix #529 - Set Redis default concurrency when value is 0

According to the README:

Use the second parameter of server.NewWorker to limit the number of concurrently running worker.Process() calls (per worker). Example: 1 will serialize task execution while 0 makes the number of concurrently executed tasks unlimited (default).

This change will set the concurrency to 2 * num CPUs when concurrency = 0. While it isn't "unlimited", it is neverthe a practical choice for most workloads (mixed CPU, I/O) and better than 1. This change should make the behavior more consistent with the doc.

Should also fix #535

+4 -2

0 comment

2 changed files

pr created time in 3 months

push eventcharleswhchan/machinery

Charles Chan

commit sha 499547fabfc25ed1eba649fe754db0c02508c9f1

Fix #529 - Set Redis default concurrency when value is 0 According to the README: > Use the second parameter of `server.NewWorker` to limit the number of concurrently running > worker.Process() calls (per worker). Example: 1 will serialize task execution while 0 makes > the number of concurrently executed tasks unlimited (default). This change will set the concurrency to 2 * num CPUs when concurrency = 0. While it isn't "unlimited", it is neverthe a practical choice for most workloads (mixed CPU, I/O) and better than 1. This change should make the behavior more consistent with the doc. Should also fix #535

view details

push time in 3 months

push eventcharleswhchan/machinery

Charles Chan

commit sha 1aaf7098c62ddbfe59e9d8661ed4151d11cd0a7f

Fix #529 - Set Redis default concurrency when value is 0 According to the README: > Use the second parameter of `server.NewWorker` to limit the number of concurrently running > worker.Process() calls (per worker). Example: 1 will serialize task execution while 0 makes > the number of concurrently executed tasks unlimited (default). This change will set the concurrency to 2 * num CPUs when concurrency = 2. While it isn't "unlimited", it is neverthe a practical choice for most workloads (mixed CPU, I/O) and better than 1. This change should make the behavior more consistent with the doc. Should also fix #535

view details

push time in 3 months

create barnchcharleswhchan/machinery

branch : issue-529-redis-default-concurrency

created branch time in 3 months

delete branch charleswhchan/machinery

delete branch : improve-readme

delete time in 3 months

pull request commentRichardKnop/machinery

Improve README

Thank you. 👍

charleswhchan

comment created time in 3 months

delete branch charleswhchan/machinery

delete branch : rename-ErrCouldNotUnmarshaTaskSignature

delete time in 3 months

delete branch charleswhchan/machinery

delete branch : fix-typos-in-redis-comments

delete time in 3 months

PR opened RichardKnop/machinery

Improve README

Minor improvement to add hyperlink to the main configuration sections.

(Probably need to split the README into multiple files later)

+25 -4

0 comment

1 changed file

pr created time in 3 months

create barnchcharleswhchan/machinery

branch : improve-readme

created branch time in 3 months

create barnchcharleswhchan/machinery

branch : rename-ErrCouldNotUnmarshaTaskSignature

created branch time in 3 months

create barnchcharleswhchan/machinery

branch : fix-typos-in-redis-comments

created branch time in 3 months

issue openedRichardKnop/machinery

[Bug] RedisConfig.NormalTasksPollPeriod does not pick up tasks if value is less than 1 second

RedisConfig.NormalTasksPollPeriod is specified as milliseconds:

// NormalTasksPollPeriod specifies the period in milliseconds when polling redis for normal tasks
// Default: 1000
NormalTasksPollPeriod int `yaml:"normal_tasks_poll_period" envconfig:"REDIS_NORMAL_TASKS_POLL_PERIOD"`

I noticed when the value is set to <1 second, Machinery does not pick up any tasks. eg. when set to 20ms, using Redis WATCH command, I notice machinery will call `blpop "machinery_tasks" "0.02".

Reading the documentation for BLPOP:

The timeout argument is interpreted as an integer value specifying the maximum number of seconds to block. A timeout of zero can be used to block indefinitely.

I did not look into where the problem is, but I suspect it is either:

  1. Redis is expecting whole integer in seconds and is returning an error with "0.02"? (see goredis:go#L327, or redis.go#L346)
  2. Problem with the go-redis library?

Workaround: Set NormalTasksPollPeriod to 1000 ms.

If sub-second timeout is not NormalTasksPollPeriod supported, then there should be a clear error message when starting machinery worker.

created time in 3 months

startedbuger/jsonparser

started time in 3 months

more