Use Third-Party Python Libraries in Hadoop Streaming

Post author: knktc
Post link: <a href="https://knktc.com/2014/05/18/mapreduce-streaming-python-library/" title="Use Third-Party Python Libraries in Hadoop Streaming">https://knktc.com/2014/05/18/mapreduce-streaming-python-library/
Copyright Notice: All articles in this blog are licensed under <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" rel="noopener" target="_blank"> BY-NC-SA unless stating additionally.

Posted on 2014-05-18 Edited on 2026-04-29

If a Hadoop Streaming Python job cannot import a third-party library, pass PYTHONPATH through -cmdenv.

When running a custom Python MapReduce job through Hadoop Streaming, you may find that the job fails if your script imports a third-party library.

When checking the logs, the failure usually turns out to be an import error because the worker cannot find that external package.

If the Python package is installed under /usr/lib/python2.6/site-packages, add the following parameter to the MapReduce command:

1	-cmdenv PYTHONPATH=$PYTHONPATH:/usr/lib/python2.6/site-packages

This passes the PYTHONPATH environment variable into the MapReduce task, so the Python interpreter can find the library and the job can run normally.