[ACCEPTED]-Determining if a given Python module is a built-in module-python-internals

sys.builtin_module_names

A tuple of strings giving the names of all 4 modules that are compiled into this Python 3 interpreter. (This information is not 2 available in any other way — modules.keys() only 1 lists the imported modules.)

If you consider it simply as asked, builtins, then 28 the accepted answer is obviously correct.

In 27 my case, I was looking for the standard 26 library as well, by which I mean a list 25 of all importable modules shipped with a 24 given Python distribution. Questions about 23 this have been asked several times but I 22 couldn't find an answer that included everything 21 I was looking for.

My use case was bucketing 20 an arbitrary x in a Python import x statement as 19 either:

• included in the Python stdlib + built-ins
• installed as a third party module
• neither

This will work for virtualenvs or 18 a global install. It queries the distribution 17 of whatever python binary is running the 16 script. The final chunk does reaches out 15 of a virtualenv, but I consider that the 14 desired behavior.

# You may need to use setuptools.distutils depending on Python distribution (from setuptools import distutils)
import distutils
import glob
import os
import pkgutil
import sys    

def get_python_library():

    # Get list of the loaded source modules on sys.path.
    modules = { 
        module
        for _, module, package in list(pkgutil.iter_modules())
        if package is False
    }

    # Glob all the 'top_level.txt' files installed under site-packages.
    site_packages = glob.iglob(os.path.join(os.path.dirname(os.__file__) 
                    + '/site-packages', '*-info', 'top_level.txt'))

    # Read the files for the import names and remove them from the modules list.
    modules -= {open(txt).read().strip() for txt in site_packages}

    # Get the system packages.
    system_modules = set(sys.builtin_module_names)

    # Get the just the top-level packages from the python install.
    python_root = distutils.sysconfig.get_python_lib(standard_lib=True)
    _, top_level_libs, _ = list(os.walk(python_root))[0]

    return sorted(top_level_libs + list(modules | system_modules))

Returns

A sorted list of imports: [..., 'imaplib', 'imghdr', 'imp', 'importlib', 'imputil', 'inspect', 'io', ...]

Explanation:

I 13 broke it up into chunks so the reason each 12 group is needed can be clear.

• modules

  • The pkgutil.iter_modules call scans all loaded modules on sys.path and returns a generator of (module_loader, name, ispkg) tuples.
  • I turn it into a set and filter out packages, since here we care only about the source modules.

• site_packages

  • Get a list of all installed packages under the conventional site-packages directory and remove them from the modules list. This roughly corresponds to the third party deps.
  • This was the hardest part to get right. Many things almost worked, like pip.get_installed_distributions or site. But pip returns the module names as they are on PyPi, not as they are when imported into a source file. Certain pathological packages would slip through the cracks, like:
    • requests-futures which is imported as requests_futures.
    • colors, which is actually ansicolors on PyPi and thus confounds any reasonable heuristic.
  • I am sure that there are certain low-usage modules that do not include the top_level.txt in their package. But this covered 100% of my use cases seems to work on everything that is correctly configured.

• system_modules

  • If you don't explicitly ask for them, you won't get these system modules, like sys, gc, errno and some other optional modules.

• top_level_libs

  • The distutils.sysconfig.get_python_lib(standard_lib=True) call returns the top-level directory of the platform independent standard library.
  • These are easy to miss because they might not live under the same python path as the other modules. If you are on OSX and running a virtualenv, these modules will actually be imported from the system install. These modules include email, logging, xml and a few more.

Conclusion

For my 2013 11 MacBookPro I found 403 modules for the python2.7 install.

   >>> print(sys.version)
   2.7.10 (default, Jul 13 2015, 12:05:58)
   [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
   >>> print(sys.hexversion)
   34015984
   >>> python_stdlib = get_python_libirary()
   >>> len(python_stdlib)
   403

I 10 put up a gist of the code and output. If you think I am missing 9 a class or have included a bogus module, I 8 would like to hear about it.

* Alternatives

• In writing this 7 post I dug around the pip and setuptools API. It is possible 6 that this information through a single module 5 but you would really need to know your way 4 around that API.

• Before I started this, I 3 was told that six has a function specifically 2 for this problem. It makes sense that might 1 exist but I couldn't find it myself.

You can use imp.is_builtin to see if a module name matches 3 a built-in module, but I can't think of 2 any way to actually introspect a module 1 object reliably.

You might also try the following:

>>> import imp
>>> f, path, desc = imp.find_module("sys")
>>> desc
('', '', 6)
>>> desc[2] == imp.C_BUILTIN
True

When you say, "built-in," do you mean, written 7 in C, or do you mean, part of the standard 6 library? If you mean the first, then looking 5 for __file__ is the right thing to do. As you can 4 see, even the Python interpreter uses the 3 presence of __file__ as an indicator of built-in-ness.

If 2 you mean "part of the standard library," then 1 it is very hard to determine.