Python gets ‘…Can’t pickle…’ when using ‘multiprocessing’.
It‘s more than ‘pickle’ issue but OS related issue.
Background
- In previous article, I bumped into the ‘lxml’ issue and the solution is to execute the function using ‘lxml’ in a separate process
- Implement a decorator to wrap a function to run in a separate process
- The ‘run_on_subprocess’ decorator uses ‘multiprocessing’ Python package
- Use Macbook for the Python service development
- The service is hosted on a Python Docker service in formal environment