Python interpreter has debug symbols but libpython.so doesn't

[manopapad]

I am trying to debug a C++ application that embeds a python interpreter, and would like to use gdb's python extensions to interpret python frames (py-bt, py-up etc.). The version of gdb I get from conda-forge (conda install gdb) is already properly configured to do this, and works for scripts executed in a standalone python interpreter. However, it doesn't work for applications that invoke the python interpreter through libpython.so, and this is likely because the version of libpython.so I get from conda-forge does not seem to contain debug symbols (see command log below). Is it possible to update the recipes to include debug symbols for libpython.so as well?

Note that I can get debug symbols on both the standalone and embedded interpreter cases if I use the debug builds from Ubuntu package python-dbg.

Commands to reproduce:

Details

prm-login:~> conda create --name temp python=3.7
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /gpfs/fs1/mpapadakis/anaconda3/envs/temp

  added / updated specs:
    - python=3.7


The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-0_gnu
  ca-certificates    conda-forge/linux-64::ca-certificates-2020.6.20-hecda079_0
  certifi            conda-forge/linux-64::certifi-2020.6.20-py37hc8dfbb8_0
  ld_impl_linux-64   conda-forge/linux-64::ld_impl_linux-64-2.34-h53a641e_7
  libffi             conda-forge/linux-64::libffi-3.2.1-he1b5a44_1007
  libgcc-ng          conda-forge/linux-64::libgcc-ng-9.2.0-h24d8f2e_2
  libgomp            conda-forge/linux-64::libgomp-9.2.0-h24d8f2e_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-9.2.0-hdf63c60_2
  ncurses            conda-forge/linux-64::ncurses-6.1-hf484d3e_1002
  openssl            conda-forge/linux-64::openssl-1.1.1g-h516909a_0
  pip                conda-forge/noarch::pip-20.1.1-py_1
  python             conda-forge/linux-64::python-3.7.6-cpython_h8356626_6
  python_abi         conda-forge/linux-64::python_abi-3.7-1_cp37m
  readline           conda-forge/linux-64::readline-8.0-h46ee950_1
  setuptools         conda-forge/linux-64::setuptools-49.2.0-py37hc8dfbb8_0
  sqlite             conda-forge/linux-64::sqlite-3.32.3-hcee41ef_1
  tk                 conda-forge/linux-64::tk-8.6.10-hed695b0_0
  wheel              conda-forge/noarch::wheel-0.34.2-py_1
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h516909a_1006


Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate temp
#
# To deactivate an active environment, use
#
#     $ conda deactivate

prm-login:~> conda activate temp
prm-login:~> cat symtest.py
import time
def foo(x):
    # give us enough time to Ctrl-C
    time.sleep(60)
    return x
def bar(x):
    return foo(x+1)
def qux(x):
    return bar(x+1)
def main():
    x = qux(0)
    print("done")
    return x
if __name__ == "__main__":
    main()
prm-login:~> cat call.c
#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pValue;
    int i;

    if (argc < 3) {
        fprintf(stderr,"Usage: call pythonfile funcname [args]\n");
        return 1;
    }

    Py_Initialize();
    pName = PyUnicode_DecodeFSDefault(argv[1]);
    /* Error checking of pName left out */

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, argv[2]);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(argc - 3);
            for (i = 0; i < argc - 3; ++i) {
                pValue = PyLong_FromLong(atoi(argv[i + 3]));
                if (!pValue) {
                    Py_DECREF(pArgs);
                    Py_DECREF(pModule);
                    fprintf(stderr, "Cannot convert argument\n");
                    return 1;
                }
                /* pValue reference stolen here: */
                PyTuple_SetItem(pArgs, i, pValue);
            }
            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            }
            else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", argv[2]);
        }
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", argv[1]);
        return 1;
    }
    if (Py_FinalizeEx() < 0) {
        return 120;
    }
    return 0;
}
prm-login:~> gcc -o call call.c -I$CONDA_PREFIX/include/python3.7m -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -ffunction-sections -pipe -isystem $CONDA_PREFIX/include -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -flto -fwrapv -g -O0 -L$CONDA_PREFIX/lib -l:libpython3.7m.so -lcrypt -lpthread -ldl  -lutil -lrt -lm
prm-login:~> gdb --args python symtest.py
GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...done.
(gdb) r
Starting program: /gpfs/fs1/mpapadakis/anaconda3/envs/temp/bin/python symtest.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
^C
Program received signal SIGINT, Interrupt.
0x00007ffff78dc0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2c80) at ../sysdeps/unix/sysv/linux/select.c:41
41	../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb) bt
#0  0x00007ffff78dc0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2c80) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x00005555556bee63 in pysleep (secs=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/timemodule.c:1843
#2  time_sleep () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/timemodule.c:371
#3  0x000055555569d72d in _PyMethodDef_RawFastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:648
#4  0x00005555556d55d0 in _PyCFunction_FastCallKeywords (kwnames=<optimized out>, nargs=<optimized out>, args=0x7ffff6c1efd0, func=0x7ffff6b3a8c0) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:734
#5  call_function () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4568
#6  0x000055555571b762 in _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3093
#7  0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6c1ee50) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547
#8  function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283
#9  _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408
#10 0x00005555557172b5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616
#11 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124
#12 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6b66050) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547
#13 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283
#14 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408
#15 0x00005555557172b5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616
#16 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124
#17 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6c20a00) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547
#18 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283
#19 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408
#20 0x00005555557172b5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616
#21 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124
#22 0x000055555568cd37 in PyEval_EvalFrameEx (throwflag=0, f=0x555555926c80) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547
#23 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:283
#24 _PyFunction_FastCallKeywords () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Objects/call.c:408
#25 0x00005555557172b5 in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic pointer>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:4616
#26 _PyEval_EvalFrameDefault () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3124
#27 0x000055555566d505 in PyEval_EvalFrameEx (throwflag=0, f=0x7ffff6b74450) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:547
#28 _PyEval_EvalCodeWithName () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3930
#29 0x000055555566e8f3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<optimized out>, globals=<optimized out>, _co=<optimized out>)
    at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:3959
#30 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/ceval.c:524
#31 0x000055555577b692 in run_mod () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/pythonrun.c:1035
#32 0x00005555557857be in PyRun_FileExFlags () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/pythonrun.c:988
#33 0x00005555557859ab in PyRun_SimpleFileExFlags () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Python/pythonrun.c:429
#34 0x0000555555786a61 in pymain_run_file (p_cf=0x7fffffff3730, filename=0x5555558be540 L"symtest.py", fp=0x555555924de0) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:428
#35 pymain_run_filename (cf=0x7fffffff3730, pymain=0x7fffffff3840) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:1607
#36 pymain_run_python (pymain=0x7fffffff3840) at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:2868
#37 pymain_main () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:3029
#38 0x0000555555786aec in _Py_UnixMain () at /home/conda/feedstock_root/build_artifacts/python_1591034797817/work/Modules/main.c:3064
#39 0x00007ffff77e6b97 in __libc_start_main (main=0x555555645ab0 <main>, argc=2, argv=0x7fffffff3998, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffff3988) at ../csu/libc-start.c:310
#40 0x000055555572b73d in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb) quit
A debugging session is active.

	Inferior 1 [process 7007] will be killed.

Quit anyway? (y or n) y
prm-login:~> PYTHONPATH=. gdb --args ./call symtest main
GNU gdb (Ubuntu 8.1-0ubuntu3.1) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./call...done.
(gdb) r
Starting program: /home/mpapadakis/call symtest main
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
^C
Program received signal SIGINT, Interrupt.
0x00007ffff698e0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2fc0) at ../sysdeps/unix/sysv/linux/select.c:41
41	../sysdeps/unix/sysv/linux/select.c: No such file or directory.
(gdb) bt
#0  0x00007ffff698e0f7 in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7fffffff2fc0) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x00007ffff7b7997a in time_sleep () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#2  0x00007ffff7bb1b5c in _PyMethodDef_RawFastCallKeywords () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#3  0x00007ffff7bb2606 in _PyCFunction_FastCallKeywords () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#4  0x00007ffff7adb3ff in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#5  0x00007ffff7ad910a in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#6  0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#7  0x00007ffff7adb116 in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#8  0x00007ffff7ad6808 in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#9  0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#10 0x00007ffff7adb116 in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#11 0x00007ffff7ad6808 in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#12 0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#13 0x00007ffff7adb116 in call_function.lto_priv () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#14 0x00007ffff7ad6808 in _PyEval_EvalFrameDefault () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#15 0x00007ffff7ada380 in function_code_fastcall () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#16 0x00007ffff7bb248f in _PyFunction_FastCallDict () from /gpfs/fs1/mpapadakis/anaconda3/envs/legate/lib/libpython3.7m.so.1.0
#17 0x00000000004010e3 in main (argc=3, argv=0x7fffffff39b8) at call.c:40
(gdb) quit
A debugging session is active.

	Inferior 1 [process 7286] will be killed.

Quit anyway? (y or n) y

Environment (conda list):

Details

$ conda list
# packages in environment at /gpfs/fs1/mpapadakis/anaconda3/envs/temp:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       0_gnu    conda-forge
ca-certificates           2020.6.20            hecda079_0    conda-forge
certifi                   2020.6.20        py37hc8dfbb8_0    conda-forge
ld_impl_linux-64          2.34                 h53a641e_7    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgomp                   9.2.0                h24d8f2e_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
pip                       20.1.1                     py_1    conda-forge
python                    3.7.6           cpython_h8356626_6    conda-forge
python_abi                3.7                     1_cp37m    conda-forge
readline                  8.0                  h46ee950_1    conda-forge
setuptools                49.2.0           py37hc8dfbb8_0    conda-forge
sqlite                    3.32.3               hcee41ef_1    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

Details about conda and system ( conda info ):

Details

$ conda info

     active environment : temp
    active env location : /gpfs/fs1/mpapadakis/anaconda3/envs/temp
            shell level : 3
       user config file : /home/mpapadakis/.condarc
 populated config files : /home/mpapadakis/.condarc
          conda version : 4.8.3
    conda-build version : 3.18.9
         python version : 3.7.4.final.0
       virtual packages : __glibc=2.27
       base environment : /gpfs/fs1/mpapadakis/anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/rapidsai-nightly/linux-64
                          https://conda.anaconda.org/rapidsai-nightly/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /gpfs/fs1/mpapadakis/anaconda3/pkgs
                          /home/mpapadakis/.conda/pkgs
       envs directories : /gpfs/fs1/mpapadakis/anaconda3/envs
                          /home/mpapadakis/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.3 requests/2.22.0 CPython/3.7.4 Linux/4.15.0-62-generic ubuntu/18.04.3 glibc/2.27
                UID:GID : 38092:30
             netrc file : None
           offline mode : False

[mingwandroid]

We don't provide debug symbols as they tend to be large and most people do not need them.

You should build the interpreter yourself. The recipe has tweak parameters for making debug builds but this stuff is new and seldom used.

I would support and effort to split debug info into separate packages for important packages though but we don't really have the tooling in place for that.

[SelvamArul]

@mingwandroid I am trying to build the python interpreter myself but build_locally.py does not support building natively on ubuntu. Could you please suggest some resources related to building the python interpreter on ubuntu?

[nikhilweee]

@mingwandroid I'm in the same boat and would like to have debug symbol support as well. Can you point us to any resources to help build the package locally?

[jakirkham]

Debug builds were added as a separate output with PR ( #597 ) and subsequent PRs

Users can install these like so

conda install conda-forge/label/python_debug::python=<version>