profile
viewpoint

Ask questions[vscode]: TypeError: cannot serialize '_io.TextIOWrapper' (pydevd_bundle.pydevd_io.IORedirector)

Environment data

  • PTVSD version: (not sure, using: ms-python.python-2019.6.24221 on vscode)
  • OS and version: macOS Mojave and Win 10
  • Python version (& distribution if applicable, e.g. Anaconda): 3.7.4 in a venv
  • Using VS Code or Visual Studio: VSCODE

Actual behavior

When I try to debug a python app with multiprocessing (pathos) and using spawn, I'm seeing this when we use pool.map: TypeError: cannot serialize '_io.TextIOWrapper' obj

I added a try except where this happens to figure which object actually causes it (since this runs fine from terminal) and I get this:

TypeError: cannot serialize '_io.TextIOWrapper' $<_pydevd_bundle.pydevd_io.IORedirector object at 0x1073aa3d0>

Expected behavior

Code should run fine as it does in terminal

Steps to reproduce:

  1. Unsure we pass pool.map(self.case_list_exec, node_args) (the case_list_exec function executes a list of test cases)
microsoft/ptvsd

Answer questions mike-pt

Hum I did original though of the logger as the problem, however this works outside of vscode without any changes.

We also attempted to comment out the logger parts so it wouldn't be passed, but perhaps we missed something, but that would be the only thing that makes sense right?

We share logging and I didn't work on that code but it should be locking and not passing that between processes (perhaps I need to look deeper into that though).

In any case here is the current stack trace after I changed the inner python module with a try except so it would print the object that can't be serialized (which feels like its something python should do).

Anyway here is the stack trace now (I can share the full code since this is private, but I think its fine to show the stack

Original:

cannot serialize '_io.TextIOWrapper' object
Traceback (most recent call last):
  File "/Users/user/gitstuff/qaenv/tools/testTools/testTrigger.py", line 52, in testing
    exitcode = execution.execution()
  File "/Users/user/gitstuff/qaenv/tools/executionTools/testExecution.py", line 55, in execution
    self.config.resultDict = self.platformExecution()
  File "/Users/user/gitstuff/qaenv/tools/executionTools/testExecution.py", line 167, in platformExecution
    driverResults = pool.map(self.__caseListExecution, nodeArgs)
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 793, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 778, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 778, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 1086, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 642, in save_reduce
    save(args)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 778, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 553, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 666, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 863, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 889, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 553, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 666, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 863, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 889, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 823, in save_list
    self._batch_appends(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 847, in _batch_appends
    save(x)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 863, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 889, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 823, in save_list
    self._batch_appends(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 847, in _batch_appends
    save(x)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 553, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 666, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 863, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 889, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 525, in save
    rv = reduce(self.proto)
TypeError: cannot serialize '_io.TextIOWrapper' object

And this is after I added the try execpt on the pickle.py code so it would print the object that can't be serialized:

TypeError: cannot serialize '_io.TextIOWrapper' $<_pydevd_bundle.pydevd_io.IORedirector object at 0x10bd8be50>
<built-in method __reduce_ex__ of IORedirector object at 0x10bd8be50> must return string or tuple
Traceback (most recent call last):
  File "/Users/user/gitstuff/qaenv/tools/testTools/testTrigger.py", line 52, in testing
    exitcode = execution.execution()
  File "/Users/user/gitstuff/qaenv/tools/executionTools/testExecution.py", line 55, in execution
    self.config.resultDict = self.platformExecution()
  File "/Users/user/gitstuff/qaenv/tools/executionTools/testExecution.py", line 167, in platformExecution
    driverResults = pool.map(self.__caseListExecution, nodeArgs)
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 794, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 779, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 779, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 1086, in save_instancemethod0
    pickler.save_reduce(MethodType, (obj.__func__, obj.__self__), obj=obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 643, in save_reduce
    save(args)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 779, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 554, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 667, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 864, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 554, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 667, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 864, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 824, in save_list
    self._batch_appends(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 848, in _batch_appends
    save(x)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 864, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 824, in save_list
    self._batch_appends(obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 848, in _batch_appends
    save(x)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 554, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 667, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/user/.virtualenvs/qaenv/lib/python3.7/site-packages/dill/_dill.py", line 910, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 864, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 890, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 545, in save
    raise PicklingError("%s must return string or tuple" % reduce)
_pickle.PicklingError: <built-in method __reduce_ex__ of IORedirector object at 0x10bd8be50> must return string or tuple
useful!
source:https://uonfu.com/
Github User Rank List