Memory leak from gRPC exporter #3796
Description
In issue #3642 , I reported memory leaks from HTTP exporters. the final resolution was that opentelemetry should not close protbuf and gnutls libraries and only the application must handle termination of those libraries as other part of the application may still use them. these are the lines I added to my application shutdown sequence that prevents all valgrind reports from the use of opentelemetry library in my code:
google::protobuf::ShutdownProtobufLibrary();
gnutls_global_deinit();
also these are for releasing opnetelmetry exporters that I call in application termination sequence:
std::shared_ptr<otl_trace::TracerProvider> none;
trace_api::Provider::SetTracerProvider(none);
now that I'm using gRPC exporter instead I got a lot of other memory leaks reports by valgrind that kinda feels like opentelmetry dose not release the grpc objects (such as Channel or Server). I'm putting a subset of valgrind reports here (very long list that dose github complains!)
==36198== 1 bytes in 1 blocks are still reachable in loss record 1 of 111
==36198== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==36198== by 0x5C361B1: gpr_malloc (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x5C366F3: gpr_strdup (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x5C394E9: grpc_core::GlobalConfigEnvString::Get() (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x4F2133A: gpr_global_config_get_grpc_dns_resolver() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x4F23CB0: grpc_core::RegisterNativeDnsResolver(grpc_core::CoreConfiguration::Builder*) (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x511F53C: grpc_core::BuildCoreConfiguration(grpc_core::CoreConfiguration::Builder*) (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x502FD86: grpc_core::CoreConfiguration::BuildNewAndMaybeSet() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x4F6F27C: grpc_channel_create (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x4CF109B: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x4CF01BC: grpc::CreateCustomChannel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc::ChannelCredentials> const&, grpc::ChannelArguments const&) (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x1E720C: opentelemetry::v1::exporter::otlp::OtlpGrpcClient::MakeChannel(opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198==
==36198== 4 bytes in 1 blocks are still reachable in loss record 2 of 111
==36198== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==36198== by 0x5C361B1: gpr_malloc (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x5C366F3: gpr_strdup (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x5C394E9: grpc_core::GlobalConfigEnvString::Get() (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x505948A: gpr_global_config_get_grpc_poll_strategy() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x5036D1F: grpc_event_engine::posix_engine::MakeDefaultPoller(grpc_event_engine::posix_engine::Scheduler*) (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x503FFC3: grpc_event_engine::experimental::PosixEnginePollerManager::PosixEnginePollerManager(std::shared_ptr<grpc_event_engine::experimental::ThreadPool>) (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x5041508: grpc_event_engine::experimental::PosixEventEngine::PosixEventEngine() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x5030012: grpc_event_engine::experimental::DefaultEventEngineFactory() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x5030074: grpc_event_engine::experimental::CreateEventEngine() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x50301A0: grpc_event_engine::experimental::GetDefaultEventEngine() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x5030458: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
...
==36198== 8 bytes in 1 blocks are still reachable in loss record 5 of 111
==36198== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==36198== by 0x4CCDC0C: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x4CCDD33: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x400571E: call_init.part.0 (dl-init.c:74)
==36198== by 0x4005823: call_init (dl-init.c:120)
==36198== by 0x4005823: _dl_init (dl-init.c:121)
==36198== by 0x401F59F: ??? (in /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
==36198==
==36198== 8 bytes in 1 blocks are still reachable in loss record 6 of 111
==36198== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==36198== by 0x4CCDE5C: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x400571E: call_init.part.0 (dl-init.c:74)
==36198== by 0x4005823: call_init (dl-init.c:120)
==36198== by 0x4005823: _dl_init (dl-init.c:121)
==36198== by 0x401F59F: ??? (in /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2)
...
==36198== 8 bytes in 1 blocks are still reachable in loss record 8 of 111
==36198== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==36198== by 0x5106C70: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x5C36070: ??? (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x5C37758: gpr_once_init (in /usr/lib/x86_64-linux-gnu/libgpr.so.29.0.0)
==36198== by 0x5106CBD: grpc_init (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x4CED053: grpc::ChannelCredentials::ChannelCredentials() (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x4CED86D: grpc::InsecureChannelCredentials() (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x1E71E8: opentelemetry::v1::exporter::otlp::OtlpGrpcClient::MakeChannel(opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1E6738: opentelemetry::v1::exporter::otlp::OtlpGrpcClient::OtlpGrpcClient(opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1ECD9F: void std::_Construct<opentelemetry::v1::exporter::otlp::OtlpGrpcClient, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&>(opentelemetry::v1::exporter::otlp::OtlpGrpcClient*, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1EC96F: std::_Sp_counted_ptr_inplace<opentelemetry::v1::exporter::otlp::OtlpGrpcClient, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&>(std::allocator<void>, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1EC613: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<opentelemetry::v1::exporter::otlp::OtlpGrpcClient, std::allocator<void>, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&>(opentelemetry::v1::exporter::otlp::OtlpGrpcClient*&, std::_Sp_alloc_shared_tag<std::allocator<void> >, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
...
==36198== 8 bytes in 1 blocks are still reachable in loss record 11 of 111
==36198== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==36198== by 0x511227E: grpc_core::RegisterTCPConnectHandshaker(grpc_core::CoreConfiguration::Builder*) (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x511F4A4: grpc_core::BuildCoreConfiguration(grpc_core::CoreConfiguration::Builder*) (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x502FD86: grpc_core::CoreConfiguration::BuildNewAndMaybeSet() (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x4F6F27C: grpc_channel_create (in /usr/lib/x86_64-linux-gnu/libgrpc.so.29.0.0)
==36198== by 0x4CF109B: ??? (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x4CF01BC: grpc::CreateCustomChannel(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<grpc::ChannelCredentials> const&, grpc::ChannelArguments const&) (in /usr/lib/x86_64-linux-gnu/libgrpc++.so.1.51.1)
==36198== by 0x1E720C: opentelemetry::v1::exporter::otlp::OtlpGrpcClient::MakeChannel(opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1E6738: opentelemetry::v1::exporter::otlp::OtlpGrpcClient::OtlpGrpcClient(opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1ECD9F: void std::_Construct<opentelemetry::v1::exporter::otlp::OtlpGrpcClient, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&>(opentelemetry::v1::exporter::otlp::OtlpGrpcClient*, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1EC96F: std::_Sp_counted_ptr_inplace<opentelemetry::v1::exporter::otlp::OtlpGrpcClient, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&>(std::allocator<void>, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)
==36198== by 0x1EC613: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<opentelemetry::v1::exporter::otlp::OtlpGrpcClient, std::allocator<void>, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&>(opentelemetry::v1::exporter::otlp::OtlpGrpcClient*&, std::_Sp_alloc_shared_tag<std::allocator<void> >, opentelemetry::v1::exporter::otlp::OtlpGrpcClientOptions const&) (in /home/unit-test/build/telemetry-cli-ut)