profile
viewpoint
Julien Desgats jdesgats @Cloudflare London, UK

cloudflare/raven-lua 94

A Lua interface to Sentry

jdesgats/ljsonschema 37

Pure Lua JSON schema validator

jdesgats/ILuaJIT 25

Readline powered shell for LuaJIT

jdesgats/lpeg 13

Optimized fork of LPeg (unofficial, WIP)

jdesgats/lpeg-serialize 9

Serialization of LPeg patterns

jdesgats/lua-lolhtml 5

Lua binding for the lol-HTML rewriter/parser

jdesgats/lua-resty-threadpool 5

Allow to use the nginx thread pools from Lua

jdesgats/lua-feedparser 2

A decent RSS and Atom XML feed parser - This fork is done for my upcoming feed aggregator

jdesgats/org.eclipse.koneki.ldt 2

Koneki project repository (koneki.ldt)

jbmvl/firefoxos-xbmc-remote 0

FirefoxOS AppDays project

issue closedjdesgats/ljsonschema

how to cache the generated validator ?

adding this as an issue because I could not find another way to reach out.

-- Note: do cache the result of schema compilation as this is a quite
-- expensive process

I read the above in the examples given, but I am not able to figure out what to cache exactly. can you please share an example of caching the compiled schema

closed time in 21 days

Mryashbhardwaj

issue commentjdesgats/ljsonschema

how to cache the generated validator ?

Yeah, one common pattern if you know the schemas in advance is to compile them at the module level:

local jsonschema = require 'jsonschema'
local _M = {} -- module exports

-- at module level
local myvalidator = jsonschema.generate_validator { ... }

function _M.some_function(obj)
    local result = myvalidator(obj)
    ...
end

return _M
Mryashbhardwaj

comment created time in 21 days

issue commentRSS-Bridge/rss-bridge

[NowOnNetflixBridge] Can't use bridge due to captcha process.

That's annoying, there's nothing much I can do in that case unfortunately. Sorry.

csisoap

comment created time in 24 days

issue commentRSS-Bridge/rss-bridge

[NowOnNetflixBridge] Can't use bridge due to captcha process.

It still works on my instance... Maybe they restrict only some countries or IPs? Can you try the following:

  • curl https://uk.newonnetflix.info/lastchance from the same machine as your instance is running
  • curl the regular new entries RSS feed: https://uk.newonnetflix.info/feed.php

Do they also return a CAPTCHA?

csisoap

comment created time in 25 days

pull request commentcloudflare/raven-lua

Support modern auth-style of Sentry DSN

Looks good, thank you for your contribution!

nevmerzhitsky

comment created time in 2 months

push eventcloudflare/raven-lua

Sergey Nevmerzhitsky

commit sha e410b388898979fba265739e02c90f7e1cf02b85

Fix a typo and a line break

view details

Sergey Nevmerzhitsky

commit sha b9f070202141dc1b955c81f54ad372b1eeb9ae89

Support DSN without a private part

view details

Sergey Nevmerzhitsky

commit sha ada30438ffcafe1249477c7318bd70836d1ce891

Add tests for the new feature

view details

push time in 2 months

PR merged cloudflare/raven-lua

Support modern auth-style of Sentry DSN

I've added support of the modern style of auth in Sentry (without private key).

I think this change don't require to create an issue.

+48 -9

0 comment

4 changed files

nevmerzhitsky

pr closed time in 2 months

push eventcloudflare/lua-resty-cookie

Ivan Babrou

commit sha 462853f1e314cdd1282c0a35c2accb9eb0d9cbf0

Sort cookie names in the test Without sorting there's no guarantee from Lua that keys will be returned in any particular order.

view details

Julien Desgats

commit sha 303e32e512defced053a6484bc0745cf9dc0d39e

Merge pull request #37 from bobrik/ivan/sort-keys Sort cookie names in the test

view details

push time in 2 months

PR merged cloudflare/lua-resty-cookie

Sort cookie names in the test

Without sorting there's no guarantee from Lua that keys will be returned in any particular order.

+8 -2

0 comment

1 changed file

bobrik

pr closed time in 2 months

issue commentcloudflare/raven-lua

Failed to send to Sentry: connection reset by peer

Hi,

Thank you for your report and sorry for the slow response. I fixed that issue using the regular ngx sender. While I'm not fundamentally opposed to a lua-resty-http backend, it adds an extra (optional) dependency but also both proposals duplicate quite a bit of code (to handle the asynchronous sending when sockets are not available) and don't pass the lint checks properly for now.

kupson

comment created time in 2 months

delete branch cloudflare/raven-lua

delete branch : fix-server-reset

delete time in 2 months

issue closedcloudflare/raven-lua

Failed to send to Sentry: connection reset by peer

The event is POSTed successfully to local Sentry instance but the ravel-lua logs following error:

2019/07/01 12:24:41 [error] 10#10: *10 recv() failed (104: Connection reset by peer), client: 127.0.0.1, server: _, request: "POST /xxx HTTP/1.1", host: "127.0.0.1:7000"
2019/07/01 12:24:41 [error] 10#10: *10 [lua] ngx.lua:32: errlog(): raven-lua failure: Failed to send to Sentry: connection reset by peer {"timestamp":"2019-07-01T12:24:41","message":"xxx",...} , client: 127.0.0.1, server: _, request: "POST /xxx HTTP/1.1", host: "127.0.0.1:7000"

Looks like it's caused by connection to server being closed by Sentry (the module sent Connection: close header in the request) in a way that generates TCP RST packets - SO_LINGER on with zero linger time followed by close().

Example packets captured by tcpdump:

12:37:28.518716 IP 172.20.6.171.34090 > 172.20.2.181.8000: Flags [S], seq 2319457555, win 27560, options [mss 1378,sackOK,TS val 922459539 ecr 0,nop,wscale 7], length 0
12:37:28.518745 IP 172.20.2.181.8000 > 172.20.6.171.34090: Flags [S.], seq 2641603589, ack 2319457556, win 27320, options [mss 1378,sackOK,TS val 119696926 ecr 922459539,nop,wscale 7], length 0
12:37:28.519025 IP 172.20.6.171.34090 > 172.20.2.181.8000: Flags [.], ack 1, win 216, options [nop,nop,TS val 922459540 ecr 119696926], length 0 
12:37:28.519078 IP 172.20.6.171.34090 > 172.20.2.181.8000: Flags [.], seq 1:1367, ack 1, win 216, options [nop,nop,TS val 922459540 ecr 119696926], length 1366
12:37:28.519111 IP 172.20.2.181.8000 > 172.20.6.171.34090: Flags [.], ack 1367, win 235, options [nop,nop,TS val 119696926 ecr 922459540], length 0
12:37:28.519216 IP 172.20.6.171.34090 > 172.20.2.181.8000: Flags [.], seq 1367:2733, ack 1, win 216, options [nop,nop,TS val 922459540 ecr 119696926], length 1366
12:37:28.519227 IP 172.20.2.181.8000 > 172.20.6.171.34090: Flags [.], ack 2733, win 257, options [nop,nop,TS val 119696926 ecr 922459540], length 0
12:37:28.519322 IP 172.20.6.171.34090 > 172.20.2.181.8000: Flags [P.], seq 2733:3812, ack 1, win 216, options [nop,nop,TS val 922459540 ecr 119696926], length 1079
12:37:28.519328 IP 172.20.2.181.8000 > 172.20.6.171.34090: Flags [.], ack 3812, win 278, options [nop,nop,TS val 119696926 ecr 922459540], length 0
12:37:28.543914 IP 172.20.2.181.8000 > 172.20.6.171.34090: Flags [P.], seq 1:367, ack 3812, win 278, options [nop,nop,TS val 119696951 ecr 922459540], length 366
12:37:28.544222 IP 172.20.6.171.34090 > 172.20.2.181.8000: Flags [.], ack 367, win 224, options [nop,nop,TS val 922459565 ecr 119696951], length 0
12:37:28.544677 IP 172.20.2.181.8000 > 172.20.6.171.34090: Flags [R.], seq 367, ack 3812, win 278, options [nop,nop,TS val 119696952 ecr 922459565], length 0 

The sock:receive('*a') function returns error on such connections.

closed time in 2 months

kupson

push eventcloudflare/raven-lua

Julien Desgats

commit sha 6c5285bcba2289952bb97a188191cc7bf933ca0a

ngx sender: try to process partial responses as normal Depending on the server/proxy configuration, it happens that a TCP reset is sent right after a successful response. When that happens, the `:read()` method returns an error with a "partial" response (which is full in this case). Now we try to process that partial buffer as normal and see if it is well formed HTTP. Fixes #30

view details

push time in 2 months

create barnchcloudflare/raven-lua

branch : fix-server-reset

created branch time in 2 months

PR opened cloudflare/tableflip

Allow the usage of a callback for Listen and ListenPacket

The ListenConfig approach added recently works well if all the connections are similar, but when different configs are required for specific connections, it quickly gets very cumbersome to use.

This new approach allows arbitrary callbacks to be used for each connection.

Note that this approach (as well as the ListenConfig one) have one drawback: if the connection customization logic changes, from one version to the next, the listening sockets will still be inherited and untouched. In this case a full service restart is necessary as the code doesn't attempt to cope with this situation.

+99 -2

0 comment

2 changed files

pr created time in 3 months

push eventcloudflare/tableflip

Julien Desgats

commit sha 446029e70a324fe995c19eda867a81eda4b8fc26

Refactor some methods to take advantage of the new `*WithCallback` methods Removes the code duplication introduced by these new methods. Now the old `Listen`/`ListenPacket` methods are just using the `WithCallback` variants under the hood.

view details

push time in 3 months

create barnchcloudflare/tableflip

branch : listen-with-callback

created branch time in 3 months

pull request commentcloudflare/raven-lua

Added lua-resty-http as sender

In ngx.lua#L170, if the queue is full the new message is dropped and an error is returned. There is no function call to process the full queue. Shouldn't there be a call to process the full queue? Originally I was looking at the lua-resty-logger-socket library as an example and in socket.lua#L533 the library will process the full queue.

I haven't called anything to process the full queue because the background task is fired every time a message is pushed. When the queue fills up, the sending task is running, but the messages are pushed faster than they can be processed. Calling it can't hurt, but it is not going to change that fact either.

In ngx.lua#L118, if the message fails to send an error is logged and the message is removed from the queue. Shouldn't it return on the log to prevent the message from being removed from the queue and lost?

Well it depends of the kind of error: if the error is caused by a temporary external error (Sentry server down, network issue, ...) retrying later might work. However if the error is caused by the message itself (wrong syntax or schema, break size limit, ...), retrying will always fail, and will result in an error loop. Unfortunately, reliably differentiate these two cases is very hard.

From my point of view Sentry is a best-effort error reporting system, it is acceptable to loose a few messages from time to time. It also deals with the case where an error is triggered at a very high rate and the server cannot ingest them. In this case dropping the messages is the only reasonable thing to do.

Also note that the luacheck test is failing, you need to add the -- luacheck: globals ngx comment at the top of the file so it knows about the ngx global.

defusevenue

comment created time in 3 months

issue closedcloudflare/raven-lua

Generated event ids are not random

I ran into a weird issue where Raven would generate the same event ids over and over again. It would also generate the same set ids in a specific order.

I resolved it by using replacing the math.random() function in util.generate_event_id() with resty.random from lua-resty-string which is bundled into Openresty.

Solution I came up with:

local resty_random = require 'resty.random'
local str = require 'resty.string'

function _M.generate_event_id()
    return str.to_hex(resty_random.bytes(16))
end

Is there an explanation to why this was happening?

closed time in 3 months

defusevenue

issue commentcloudflare/raven-lua

Generated event ids are not random

This is because the PRNG needs to be seeded with math.randomseed, otherwise you will always use the default seed that produce the same suite of values. This is outside of the scope of this library and should be done globally during initialization.

defusevenue

comment created time in 3 months

Pull request review commentcloudflare/raven-lua

Added lua-resty-http as sender

+-- vim: st=4 sts=4 sw=4 et:+--- Network backend using [lua-resty-http](https://github.com/ledgetech/lua-resty-http).+--- Supports https, http, and keepalive for better performance.+--+-- @module raven.senders.lua-resty-http+-- @copyright 2014-2017 CloudFlare, Inc.+-- @license BSD 3-clause (see LICENSE file)++local util = require 'raven.util'+local http = require 'resty.http'+local cjson = require 'cjson'++local tostring = tostring+local setmetatable = setmetatable+local table_concat = table.concat+local parse_dsn = util.parse_dsn+local generate_auth_header = util.generate_auth_header+local _VERSION = util._VERSION+local _M = {}++local mt = {}+mt.__index = mt++function mt:send(json_str)+    local httpc = http.new()+    local res, err = httpc:request_uri(self.server, {+        method = "POST",+        headers = {+            ['Content-Type'] = 'applicaion/json',+            ['User-Agent'] = "raven-lua-http/" .. _VERSION,+            ['X-Sentry-Auth'] = generate_auth_header(self),+            ["Content-Length"] = tostring(#json_str),+        },+        body = cjson.encode(json_str),+        keepalive = self.opts.keepalive,+        keepalive_timeout = self.opts.keepalive_timeout,+        keepalive_pool = self.opts.keepalive_pool+    })++    if not res then+        return nil, table_concat(err)+    end++    return true

In this case the server responds a non-200 answer, the error would be silently ignored here. The raven library attempts to write the message in the error log as a last resort in case the sender cannot send it (https://github.com/cloudflare/raven-lua/blob/master/raven/init.lua#L323).

defusevenue

comment created time in 3 months

Pull request review commentcloudflare/raven-lua

Added lua-resty-http as sender

+-- vim: st=4 sts=4 sw=4 et:+--- Network backend using [lua-resty-http](https://github.com/ledgetech/lua-resty-http).+--- Supports https, http, and keepalive for better performance.+--+-- @module raven.senders.lua-resty-http+-- @copyright 2014-2017 CloudFlare, Inc.+-- @license BSD 3-clause (see LICENSE file)++local util = require 'raven.util'+local http = require 'resty.http'+local cjson = require 'cjson'++local tostring = tostring+local setmetatable = setmetatable+local table_concat = table.concat+local parse_dsn = util.parse_dsn+local generate_auth_header = util.generate_auth_header+local _VERSION = util._VERSION+local _M = {}++local mt = {}+mt.__index = mt++function mt:send(json_str)+    local httpc = http.new()+    local res, err = httpc:request_uri(self.server, {

Note that sockets are not always available with lua_nginx_module: some phases such as init, set and log don't support async operations. The plain ngx module works around this limitation by spawning a timer that does the actual sending.

Perhaps there is a way to extract that logic so it doesn't need to be replicated here.

defusevenue

comment created time in 3 months

Pull request review commentcloudflare/raven-lua

Added lua-resty-http as sender

 to Sentry.]], dependencies = {   "lua >= 5.1",   "lua-cjson",+  "lua-resty-http"

I usually not add senders dependencies on the rockspec because it doesn't make sense to every user of the module to install them. This is why luasocket and luasec are not listed here as they are specific to the luasocket backend.

defusevenue

comment created time in 3 months

Pull request review commentcloudflare/raven-lua

Added lua-resty-http as sender

+-- vim: st=4 sts=4 sw=4 et:+--- Network backend using [lua-resty-http](https://github.com/ledgetech/lua-resty-http).+--- Supports https, http, and keepalive for better performance.+--+-- @module raven.senders.lua-resty-http+-- @copyright 2014-2017 CloudFlare, Inc.+-- @license BSD 3-clause (see LICENSE file)++local util = require 'raven.util'+local http = require 'resty.http'+local cjson = require 'cjson'++local tostring = tostring+local setmetatable = setmetatable+local table_concat = table.concat+local parse_dsn = util.parse_dsn+local generate_auth_header = util.generate_auth_header+local _VERSION = util._VERSION+local _M = {}++local mt = {}+mt.__index = mt++function mt:send(json_str)+    local httpc = http.new()+    local res, err = httpc:request_uri(self.server, {+        method = "POST",+        headers = {+            ['Content-Type'] = 'applicaion/json',+            ['User-Agent'] = "raven-lua-http/" .. _VERSION,+            ['X-Sentry-Auth'] = generate_auth_header(self),+            ["Content-Length"] = tostring(#json_str),+        },+        body = cjson.encode(json_str),+        keepalive = self.opts.keepalive,+        keepalive_timeout = self.opts.keepalive_timeout,+        keepalive_pool = self.opts.keepalive_pool+    })++    if not res then+        return nil, table_concat(err)

From my understanding request_uri returns a string error directly, not a table. Attempting to concatenate it will throw an error.

defusevenue

comment created time in 3 months

more