profile
viewpoint
Luis Hernandez colmex AIS Toronto

colmex/frontera_example 10

Example frontera project

colmex/charts 0

Curated applications for Kubernetes

colmex/ellsberg 0

Makes sure that your team on slack knows about critical errors in your ELK stack

colmex/gcsfuse 0

A user-space file system for interacting with Google Cloud Storage

colmex/mockrdd 0

A Python3 module for testing PySpark code

colmex/murphy-ml-exercises-solutions 0

Exercises and Solutions to Murphy's Book

colmex/python-json-logger 0

Json Formatter for the standard python logger

colmex/redux-auth-wrapper 0

A React Higher Order Component (HOC) for handling Authentication and Authorization with Routing and Redux

issue openedsqueaky-pl/japronto

我把japronto移植到了raspberry pi,然后用wrk压力测试。

image

The code is from example async.py 。 I don't know why asynchronous functions are so inefficient on raspberry, while synchronous functions are more efficient than my PC.

created time in a month

pull request commentsqueaky-pl/japronto

Small improvments

Does this code have any chances to be merged?

proggga

comment created time in a month

issue commentsqueaky-pl/japronto

Is this project dead?

When will we can use it?

jorgerpo

comment created time in a month

issue commentsqueaky-pl/japronto

After some number of requests, the sever prints 'incomplete body'.

text expect a string not byte. So Request.Response can't return. u can use r.text instead of r.body.

stuz5000

comment created time in a month

issue commentsqueaky-pl/japronto

i used aiomysql with singleton mode

I see!I always thought the lock would be created in children process,It is actually created when the class is defined. I use asyncio.Lock and succeed. If using threading.Lock,the children process will blocking themselves, because DbUtil._instance = await aiomysql.create_pool.

junbaibai0719

comment created time in 2 months

issue commentsqueaky-pl/japronto

i used aiomysql with singleton mode

I didn't run the code but this happens because this most probably happens because _instance_lock = threading.Lock() gets created in the master server process and then gets shared between all the spawned children processes.

You can try removing with DbUtil._instance_loc and see what happens.

junbaibai0719

comment created time in 2 months

issue openedsqueaky-pl/japronto

i used aiomysql with singleton mode

when i set multi process, only one run.

code

import aiomysql
import threading

class DbUtil(object):
    _instance_lock = threading.Lock()
    def __init_(self):
        pass

    async def __new__(cls, *args, **kwargs):
        if not hasattr(DbUtil, "_instance"):
            with DbUtil._instance_lock:
                if not hasattr(DbUtil, "_instance"):
                    DbUtil._instance = await aiomysql.create_pool(host='127.0.0.1', port=3306,
                           user='root', password='ll980810',
                           db='mysql')            
        return DbUtil._instance

from japronto import Application
import aiomysql
import asyncio
import traceback
import os

async def hello(request):
    res = "hello world"
    try:
        pool = await DbUtil()
        print(os.getpid())
        async with pool.acquire() as conn:
            async with conn.cursor() as cur:
                await cur.execute("SELECT * from user;")
                data = await cur.fetchone()
                res = data[1]
    except Exception as e:
        print(e)
        traceback.print_exc()
    return request.Response(text=res)


def index(request):
    print(os.getpid())
    return request.Response(text="hello world")
    
    
app = Application()

app.router.add_route('/hello', hello)
app.router.add_route('/', index)

app.run(port=8081, worker_num=6, debug=False)

while i request 127.0.0.1:8081/, proccessor id

27719 27716 27710 27722 27719 27716 27710 27722 27719 27710 27716 27722 27719 27710 27716

while i request 127.0.0.1:8081/hello, proccessor id

27711 27711 27711 27711 27711 27711 27711 27711 27711 27711 27711 27711 27711 27711 27711 27711

created time in 2 months

issue closedsqueaky-pl/japronto

Request - Benchmarks - Add Nim/Jester

import asyncdispatch, jester, os, strutils

routes:
  get "/":
    resp "Hello world!"

build with: nim c --threads:on --gc:markAndSweep -d:release {file.nim}

On my test machine, I get 771768 requests / second.

For comparison, I get 974294 requests / second with japronto on the same machine.

closed time in 3 months

UNIcodeX

issue openedcolmex/frontera_example

[ENCODING] unpack error

Getting following error for reading messages from kafka

2020-09-07 18:51:39 [messagebus-backend] WARNING: Could not decode message: b'http://quotes.toscrape.com/page/1/', error unpack(b) received extra data.

created time in 3 months

issue openedcolmex/frontera_example

keyerror: Frontier-Request

Getting keyerror frontier-request. Probably due to feeding urls to spider directly and not through frontera

created time in 3 months

more