Python ???multiprocessing??? ???Can???t pickle??????

Python gets ‘…Can’t pickle…’ when using ‘multiprocessing’.
It‘s more than ‘pickle’ issue but OS related issue.

Background

  • In previous article, I bumped into the ‘lxml’ issue and the solution is to execute the function using ‘lxml’ in a separate process
  • Implement a decorator to wrap a function to run in a separate process
  • The ‘run_on_subprocess’ decorator uses ‘multiprocessing’ Python package
  • Use Macbook for the Python service development
  • The service is hosted on a Python Docker service in formal environment

Website