profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/haberman/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

google/bloaty 3380

Bloaty McBloatface: a size profiler for binaries

protocolbuffers/upb 1206

a small protobuf implementation in C

haberman/jitdemo 201

Demo programs for my blog entry "Hello, JIT World!"

haberman/gazelle 162

A system for creating fast, reusable parsers

haberman/upb 58

a small protobuf implementation in C

haberman/dumpfp 50

A small little tool for dumping a floating-point number in its native format

haberman/vtparse 48

A library to parse terminal escape sequences exactly how the real hardware does.

haberman/dblbook 26

Double-Booked: a Chrome app for double-entry accounting.

haberman/fast-recs-collate 8

A fast version of recs-collate

PR opened protocolbuffers/protobuf

Performance fix for DynamicMessage: force GetRaw() to be inlined.

Fixes: https://github.com/protocolbuffers/protobuf/issues/9014

+1 -1

0 comment

1 changed file

pr created time in 15 hours

create barnchhaberman/protobuf

branch : dyn-msg-perf-fix

created branch time in 15 hours

issue commentprotocolbuffers/protobuf

Performance regressions in Python in version 3.18.0

I was able to reproduce this, thanks.

Here are profiles before and after. In 3.18.0 we have a costly DynamicMessage::MutableRaw() call showing up that we didn't see before. This accounts for upwards of 17% of the CPU, which closely matches the ~20% regression we are seeing.

3.17.3:

  21.69%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::DynamicMessage::SharedCtor                                                                              
  18.13%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::DynamicMessage::~DynamicMessage                                                                         
   3.85%  python   libc-2.31.so                             [.] malloc_consolidate                                                                                                        
   2.99%  python   libc-2.31.so                             [.] _int_malloc                                                                                                               
   2.49%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::internal::ArenaStringPtr::Destroy                                                                       
   2.23%  python   [kernel.kallsyms]                        [k] clear_page_erms                                                                                                           
   2.02%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] 0x00000000000f6681                                                                                                        
   1.86%  python   libc-2.31.so                             [.] _int_free                                                                                                                 
   1.80%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::internal::WireFormat::_InternalParseAndMergeField  

3.18.0:

  17.57%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::DynamicMessage::SharedCtor                                                                              
  16.99%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::DynamicMessage::MutableRaw                                                                              
  15.36%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::DynamicMessage::~DynamicMessage                                                                         
   2.71%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::internal::ArenaStringPtr::Destroy                                                                       
   2.62%  python   libc-2.31.so                             [.] _int_malloc                                                                                                               
   2.26%  python   _message.cpython-39-x86_64-linux-gnu.so  [.] google::protobuf::Descriptor::FindFieldByNumber                                                                           
   1.97%  python   [kernel.kallsyms]                        [k] clear_page_erms                                                                                                           
   1.87%  python   libc-2.31.so                             [.] _int_free
BenRKarl

comment created time in 17 hours

PullRequestReviewEvent

issue commentprotocolbuffers/protobuf

Faster Python Protobuf implementation?

We're aiming to have something in prerelease in the next few months.

If you're feeling adventurous, you can try it out from my development branch now. It is not 100% complete, but most APIs are implemented and passing tests. The performance should give you a rough ballpark of what to expect.

To try it out:

  1. install the Python protobuf package from the master branch of protobuf/python.
  2. compile your Python protos with a protoc built from the master branch of protobuf. You'll know you did this correctly if your _pb2.py files contain the string AddSerializedFile.
  3. Build the C extension from my development branch using Bazel:
$ git clone --branch python https://github.com/haberman/upb.git
$ cd upb
$ bazel build -c opt python:message_ext python:api_implementation_ext
$ ls bazel-bin/python/google/protobuf/**/*.so
bazel-bin/python/google/protobuf/internal/_api_implementation.cpython-39-x86_64-linux-gnu.so
bazel-bin/python/google/protobuf/pyext/_message.cpython-39-x86_64-linux-gnu.so

Copy these .so's to your PYTHONPATH, such that they can be loaded as google.protobuf.internal._api_implementation and google.protobuf.pyext._message, respectively.

If you have any feedback it could be useful in helping to gauge the performance improvement vs. the existing C++ extension. It could also help point to any places where we still need to optimize further.

danbosnichill

comment created time in a day

issue commentprotocolbuffers/protobuf

Performance regressions in Python in version 3.18.0

@anandolee and I have been looking into this and we've had trouble reproducing this regression.

I modified your script a bit to use timeit() to run the operation repeatedly, to try and get a number with fewer decimal places, and to try to reduce noise/variance in the result.

from protobuf_fixture_pb2 import ProtobufFixture
import google.protobuf as pb
import timeit

def BM(func):
  timer = timeit.Timer(func, globals=globals())
  iters, time = timer.autorange()
  ns_per_iter = time / iters * (10 ** 9)
  return f"{int(ns_per_iter)} ns/iter"

msg = ProtobufFixture()
create = BM('msg = ProtobufFixture()')
assign = BM('msg.name = "Test"')
print(f"Version: {pb.__version__}, create: {create}, assign: {assign}")

I tried this with 3.18.0 and 3.17.3, with both the pure-Python and Python/C++ implementations. My results were:

# 3.18.0
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python test.py 
Version: 3.18.0, create: 141 ns/iter, assign: 121 ns/iter
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python test.py 
Version: 3.18.0, create: 962 ns/iter, assign: 408 ns/iter

# 3.17.3
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp python test.py 
Version: 3.17.3, create: 137 ns/iter, assign: 114 ns/iter
$ PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python python test.py 
Version: 3.17.3, create: 921 ns/iter, assign: 434 ns/iter

I'm not sure why we would be getting different results from you. What results do you get from my script?

BenRKarl

comment created time in 2 days

issue commentprotocolbuffers/protobuf

Performance regressions in Python in version 3.18.0

@BenRKarl could you include the code that is using ProtobufFixture in your second example, so we have a full repro of the regression that uses protobuf only?

BenRKarl

comment created time in 2 days

push eventhaberman/upb

Joshua Haberman

commit sha d1862a46b0affef426e588955277c4a6c2001321

Implemented MessageSet!

view details

push time in 2 days

push eventhaberman/upb

Joshua Haberman

commit sha 9e85f1c3f5b0f56727af0abab5735819835f6e1a

Small correctness fix for ClearField().

view details

push time in 2 days

push eventhaberman/upb

Joshua Haberman

commit sha 4d35e29faee83b2f04473c982a9bcbe226b31e5a

Implemented required field handling.

view details

push time in 2 days

push eventhaberman/upb

Joshua Haberman

commit sha 871a67e165ef6756d7c05e929777beb35df6159a

Fixes for Python 2.7.

view details

push time in 3 days

PullRequestReviewEvent

push eventhaberman/upb

Joshua Haberman

commit sha 60df4cb32434916846eb5198b5ac352804695d3c

Fixed several compiler warnings generated by GCC.

view details

push time in 4 days

pull request commentprotocolbuffers/protobuf

Update ruby_generator.cc to allow proto2 imports in proto3

This looks great. Could you also add a test in one of these protos, to ensure that the generated code can be successfully loaded into a Ruby process and used?

https://github.com/protocolbuffers/protobuf/tree/master/ruby/tests

I think this is a good candidate; you could make this file import one of the proto2 protos from this directory: https://github.com/protocolbuffers/protobuf/blob/master/ruby/tests/basic_test.proto

Then add a test in https://github.com/protocolbuffers/protobuf/blob/master/ruby/tests/basic.rb

zhangskz

comment created time in 4 days

push eventhaberman/upb

Joshua Haberman

commit sha 9b1aa3e2cfb1e450f4114a02c75ebed677c2a322

More encode optimization.

view details

push time in 4 days

push eventhaberman/upb

Joshua Haberman

commit sha b5c04255d37c23240452dac66eaed85ca1fd7bb0

Encoding optimization, speeds up encoder by 36%.

view details

push time in 4 days

push eventhaberman/upb

Joshua Haberman

commit sha d4ff5724dcc76f59e940b2b17cb377ad224f4235

Optimization for checking enum fields.

view details

Joshua Haberman

commit sha 50dd4fafaaef0e33af3125d6cb91417e4ed76cde

A few optimizations for the decoder.

view details

Joshua Haberman

commit sha 70965b65fc581cc209ca744a81540bc30595fa7e

A few optimizations.

view details

push time in 4 days

push eventhaberman/upb

Joshua Haberman

commit sha 7ba22ddccc7eea1f692a82666c24c6a083c03645

Annoying proto2 enum behavior is implemented!

view details

push time in 5 days

PullRequestReviewEvent

Pull request review commentgoogleapis/google-cloud-ruby

feat: add smoketest for enum values handling in compute

+require "minitest/autorun"++require "google/cloud/errors"+require "google/cloud/compute/v1/accelerator_types"++# Misc smoke tests for Compute Client+class PaginationSmokeTest < Minitest::Test+  def setup+    @default_zone = "us-central1-a"+    @default_project = ENV["COMPUTE_TEST_PROJECT"]+    @client = Google::Cloud::Compute::V1::AcceleratorTypes::Rest::Client.new+    skip "COMPUTE_TEST_PROJECT must be set before running this test" if @default_project.nil?+  end++  def test_parse_unknown_enum_value+    deprectation_status_json = Google::Cloud::Compute::V1::DeprecationStatus.new(+      deleted: "foo",+      state: :OBSOLETE).to_json++    ds_decoded = ::Google::Cloud::Compute::V1::DeprecationStatus.decode_json deprectation_status_json+    assert_equal :OBSOLETE, ds_decoded.state++    unknown_enum_json = deprectation_status_json.gsub "OBSOLETE", "THIS_VALUE_DOES_NOT_EXIST"+    ds_decoded = ::Google::Cloud::Compute::V1::DeprecationStatus.decode_json unknown_enum_json

I think you need to pass {:ignore_unknown_fields => true} to get the ignoring behavior: https://github.com/protocolbuffers/protobuf/blob/master/ruby/ext/google/protobuf_c/message.c#L953-L954

viacheslav-rostovtsev

comment created time in 5 days

push eventhaberman/upb

Joshua Haberman

commit sha 70f5e7ffe2be6dac998e73ecddcf52c1f2394cc6

WIP.

view details

Joshua Haberman

commit sha 5bddf71852e3c0da9b06a2a0a2a03269bd120bcc

proto2 enum support partially done.

view details

push time in 5 days

issue commentbazelbuild/bazel

Please add a way to control compilation mode in host-build tools

Please don't take away --nodistinct_host_configuration. As this bug illustrates, it's an invaluable tool for debugging genrules.

Even if the situation isn't as gnarly as what @jmillikin-stripe ran into, --nodistinct_host_configuration is still much easier than:

  1. bazel build -s
  2. watch millions of log lines fly by, try to find precisely the command you care about
  3. copy and paste the command, hoping that the binaries in bazel-bin are the debug ones
jmillikin-stripe

comment created time in 5 days

push eventhaberman/upb

Joshua Haberman

commit sha 833016d9ab689ad6caf5886907c1325b790f417d

Disable -fvisibility=hidden. It doesn't seem to be working with Python 3.7.2.

view details

push time in 6 days

push eventhaberman/upb

Joshua Haberman

commit sha ad775b78e64b6e3d839449e0ef509d63629e4db3

Fixed opt build.

view details

push time in 6 days

push eventhaberman/upb

Joshua Haberman

commit sha ae1c742bff49c640e30ba50bd218a6fff06690f6

Moved list of layouts from .upbdefs.c to .upb.c.

view details

Joshua Haberman

commit sha c9884aad1b2259f1711a8579b3703cd89b56fab5

WIP.

view details

Joshua Haberman

commit sha a52fb799652fa7ac9546779ec1c6eb8f64715a4d

First version of a new design doc.

view details

Joshua Haberman

commit sha 975ea595f894ef2a79917101f27e1491a448bf2c

Fleshed out DESIGN.md a bit more.

view details

Joshua Haberman

commit sha 982f26aad42291064878ff64cb5a43d69723f91c

Merge pull request #418 from haberman/design-doc First version of a new design doc.

view details

Joshua Haberman

commit sha 4e5503da358d5b3008609138367eaae6a8896c19

Options everywhere!

view details

Joshua Haberman

commit sha 3c448429df3bf9fd1d90ba11edb5d397d48343b3

Fixed a bug in symtab.

view details

Joshua Haberman

commit sha 84f7e83486cc508ffb1b37fcf749b1b96d56bb7c

WIP.

view details

Joshua Haberman

commit sha fd77d1edbbc39261d5a0f9773fabfe8f5dfd76f7

Merge branch 'master' into python

view details

Joshua Haberman

commit sha 742509e16b63edac97822d135f42e4ee800b8c43

CopyToProto.

view details

push time in 6 days

push eventhaberman/protobuf

Joshua Haberman

commit sha 745b8ca93b5ea550a1ddc613c6808ec1fc503ddd

One last try.

view details

push time in 6 days

push eventhaberman/protobuf

Joshua Haberman

commit sha 07c60252fff5aa851c66e3b9163479052a7b6911

Oops, fix it.

view details

push time in 6 days

push eventhaberman/protobuf

Joshua Haberman

commit sha db521d9612b3e8cc486444323af2a30cfa823512

Don't add the same file repeatedly.

view details

push time in 6 days

push eventprotocolbuffers/upb

Joshua Haberman

commit sha a52fb799652fa7ac9546779ec1c6eb8f64715a4d

First version of a new design doc.

view details

Joshua Haberman

commit sha 975ea595f894ef2a79917101f27e1491a448bf2c

Fleshed out DESIGN.md a bit more.

view details

Joshua Haberman

commit sha 982f26aad42291064878ff64cb5a43d69723f91c

Merge pull request #418 from haberman/design-doc First version of a new design doc.

view details

push time in 9 days

PR merged protocolbuffers/upb

First version of a new design doc.
+206 -0

0 comment

2 changed files

haberman

pr closed time in 9 days