Discussion:
configure 'ncurses' when it is set '-flto' in CFLAGS, CXXFLAGS and LDFLAGS
Denis Pronin
2018-10-03 20:20:15 UTC
Permalink
Hi,

I work on gentoo linux, where CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS are
exposed to ./configure script of ncurses library. As 'configure' goes
it checks whether it is supported linkage to dl*() functions. CFLAGS,
CXXFLAGS and LDFLAGS contains '-flto' flag standing for 'link time
optimization' taking place when compiling and linking against libraries
and object files

./configure verifies support of dl*() functions by compiling the FIRST
conftest.c looking like:

__________________

#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C"
#endif
/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char dlsym();
char (*f)();

int
main (void)
{
/* The GNU C library defines this for functions which it implements
to always fail with ENOSYS. Some functions are actually named
something starting with __ and the normal name is an alias. */
#if defined (__stub_dlsym) || defined (__stub___dlsym)
choke me
#else
f = dlsym; /* workaround for ICC 12.0.3 */ if (f == 0) return 1;
#endif

;
return 0;
}


_____________

If we compile this without '-flto' flag provided to a compiler and a
linker we get failed since 'dlsym' is required to exist since no
optimization is taken and applied. This definitely leads the configure
script to checking if the symbols are available in libdl library and
./configure tries to compile the other conftest.c by calling a compiler
and a linker with the same flags appended by -ldl, the SECOND
conftest.c looks like:

____

#include "confdefs.h"
#include <dlfcn.h>
int
main (void)
{

void *obj;
if ((obj = dlopen("filename", 0)) != 0) {
if (dlsym(obj, "symbolname") == 0) {
dlclose(obj);
}
}
;
return 0;
}

___________

And it gets succeeded in comping this program having been linked
against -ldl


The issue is: if we compile the FIRST conftest.c with supplemented flag
'-flto' the conftest.c passes fairly because when 'lto' is enabled it
erases all the symbols not participating in any invocations. Since
'dlsym' is not used, only is assigned, it is deleted from binary upon
linking. As you might see in contrast to the version, where 'lto' is
disabled, it gets successfully compiled and afterwards fails to compile
some tests being deceived that 'dlsym' is available without -ldl linked

The reason is '-flto' that makes a compiler behave wisely with unused
and prone-to-optimized-out variables and calls

What I could suggest you is adding one single line with usage of
'dlsym' by calling it in the FIRST conftest.c to prevent '-flto' from
displacing 'dlsym' from the object file when it is a linker's state of
preparing a binary

Just make the FIRST conftest.c be:

__________________

#include <assert.h>
/* Override any gcc2 internal prototype to avoid an error. */
#ifdef __cplusplus
extern "C"
#endif
/* We use char because int might match the return type of a gcc2
builtin and then its argument prototype would still apply. */
char dlsym();
char (*f)();

int
main (void)
{
/* The GNU C library defines this for functions which it implements
to always fail with ENOSYS. Some functions are actually named
something starting with __ and the normal name is an alias. */
#if defined (__stub_dlsym) || defined (__stub___dlsym)
choke me
#else
f = dlsym; /* workaround for ICC 12.0.3 */ if (f == 0) return 1;
f(NULL, NULL);
#endif
;
return 0;
}


_____________

It is the most simple fix I could advise

Thank you
Thomas Dickey
2018-10-04 00:12:27 UTC
Permalink
Post by Denis Pronin
Hi,
I work on gentoo linux, where CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS are
exposed to ./configure script of ncurses library. As 'configure' goes
it checks whether it is supported linkage to dl*() functions. CFLAGS,
CXXFLAGS and LDFLAGS contains '-flto' flag standing for 'link time
optimization' taking place when compiling and linking against libraries
and object files
./configure verifies support of dl*() functions by compiling the FIRST
thanks (I'll investigate - the discussion in the bug report was not useful)
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Denis Pronin
2018-10-04 07:46:57 UTC
Permalink
I am sorry. Something weird happened when I was sending to you an email, my email client had unexpectedly quit and I got confused whether the email was sent; I resent you the other email where briefly told you what's been going on. There a link has been, take a look

As I understood it would be nice if you regenerate configure with a fresh version of 'autoconf'
Post by Thomas Dickey
 Hi,
 I work on gentoo linux, where CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS are
 exposed to ./configure script of ncurses library. As 'configure' goes
 it checks whether it is supported linkage to dl*() functions. CFLAGS,
 CXXFLAGS and LDFLAGS contains '-flto' flag standing for 'link time
 optimization' taking place when compiling and linking against libraries
 and object files
 ./configure verifies support of dl*() functions by compiling the FIRST
thanks (I'll investigate - the discussion in the bug report was not useful)
--
https://invisible-island.net
ftp://ftp.invisible-island.net
Thomas Dickey
2018-10-04 08:18:13 UTC
Permalink
I am sorry. Something weird happened when I was sending to you an email, my
email client had unexpectedly quit and I got confused whether the email was
sent; I resent you the other email where briefly told you what's been going
on. There a link has been, take a look
I saw it.
As I understood it would be nice if you regenerate configure with a fresh
version of 'autoconf'
that would be unlikely to solve this problem: at the time, I compared against
"fresh" autoconf, and its stub code did not fix the configuration that I was
interested in, nor did it improve a few related problems with weak symbols
on other platforms.

I'll take both into account (thanks)
Post by Thomas Dickey
 Hi,
 I work on gentoo linux, where CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS are
 exposed to ./configure script of ncurses library. As 'configure' goes
 it checks whether it is supported linkage to dl*() functions. CFLAGS,
 CXXFLAGS and LDFLAGS contains '-flto' flag standing for 'link time
 optimization' taking place when compiling and linking against libraries
 and object files
 ./configure verifies support of dl*() functions by compiling the FIRST
thanks (I'll investigate - the discussion in the bug report was not useful)
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Denis Pronin
2018-10-04 08:24:36 UTC
Permalink
maybe I could suggest you a patch to the 'configure' script? or you will do it yourself?
I would suggest that you should somehow use 'dlsym' in the first conftest.c, correctly, with the right interface of dlsym, from dlfct.h. This shall prevent the linker, in 'flto' mode, from throwing 'dlsym' away for it not having been used
Post by Thomas Dickey
 I am sorry. Something weird happened when I was sending to you an email, my
 email client had unexpectedly quit and I got confused whether the email was
 sent; I resent you the other email where briefly told you what's been going
 on. There a link has been, take a look
I saw it.
 As I understood it would be nice if you regenerate configure with a fresh
 version of 'autoconf'
that would be unlikely to solve this problem: at the time, I compared against
"fresh" autoconf, and its stub code did not fix the configuration that I was
interested in, nor did it improve a few related problems with weak symbols
on other platforms.
I'll take both into account (thanks)
 >>  Hi,
 >>
 >>  I work on gentoo linux, where CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS are
 >>  exposed to ./configure script of ncurses library. As 'configure' goes
 >>  it checks whether it is supported linkage to dl*() functions. CFLAGS,
 >>  CXXFLAGS and LDFLAGS contains '-flto' flag standing for 'link time
 >>  optimization' taking place when compiling and linking against libraries
 >>  and object files
 >>
 >>  ./configure verifies support of dl*() functions by compiling the FIRST
 >
 > thanks (I'll investigate - the discussion in the bug report was not useful)
--
https://invisible-island.net
ftp://ftp.invisible-island.net
Denis Pronin
2018-10-04 08:30:22 UTC
Permalink
Definitely, we should reach out to the community of 'autotools' developers, as autoconf generates incorrect conftest.c, shouldn't we?
I would suggest them how to resolve the issue
Post by Denis Pronin
maybe I could suggest you a patch to the 'configure' script? or you will do it yourself?
I would suggest that you should somehow use 'dlsym' in the first conftest.c, correctly, with the right interface of dlsym, from dlfct.h. This shall prevent the linker, in 'flto' mode, from throwing 'dlsym' away for it not having been used
  I am sorry. Something weird happened when I was sending to you an email, my
  email client had unexpectedly quit and I got confused whether the email was
  sent; I resent you the other email where briefly told you what's been going
  on. There a link has been, take a look
 I saw it.
  As I understood it would be nice if you regenerate configure with a fresh
  version of 'autoconf'
 that would be unlikely to solve this problem: at the time, I compared against
 "fresh" autoconf, and its stub code did not fix the configuration that I was
 interested in, nor did it improve a few related problems with weak symbols
 on other platforms.
 I'll take both into account (thanks)
  >>  Hi,
  >>
  >>  I work on gentoo linux, where CC, CXX, CFLAGS, CXXFLAGS, LDFLAGS are
  >>  exposed to ./configure script of ncurses library. As 'configure' goes
  >>  it checks whether it is supported linkage to dl*() functions. CFLAGS,
  >>  CXXFLAGS and LDFLAGS contains '-flto' flag standing for 'link time
  >>  optimization' taking place when compiling and linking against libraries
  >>  and object files
  >>
  >>  ./configure verifies support of dl*() functions by compiling the FIRST
  >
  > thanks (I'll investigate - the discussion in the bug report was not useful)
 --
 https://invisible-island.net
 ftp://ftp.invisible-island.net
Thomas Dickey
2018-10-05 09:05:54 UTC
Permalink
Post by Denis Pronin
Definitely, we should reach out to the community of 'autotools' developers,
as autoconf generates incorrect conftest.c, shouldn't we?
I would suggest them how to resolve the issue
I'm not going to do that (thanks).
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Thomas Dickey
2018-10-05 09:08:03 UTC
Permalink
Post by Denis Pronin
maybe I could suggest you a patch to the 'configure' script? or you will do it yourself?
I would suggest that you should somehow use 'dlsym' in the first conftest.c, correctly, with the right interface of dlsym, from dlfct.h. This shall prevent the linker, in 'flto' mode, from throwing 'dlsym' away for it not having been used
yes ("somehow").

I will summarize the details in the program's respective changelogs.
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Thomas Dickey
2018-10-07 01:39:08 UTC
Permalink
Post by Thomas Dickey
Post by Denis Pronin
maybe I could suggest you a patch to the 'configure' script? or you will do it yourself?
I would suggest that you should somehow use 'dlsym' in the first conftest.c, correctly, with the right interface of dlsym, from dlfct.h. This shall prevent the linker, in 'flto' mode, from throwing 'dlsym' away for it not having been used
yes ("somehow").
I will summarize the details in the program's respective changelogs.
done with that:

https://invisible-island.net/autoconf/autoconf-252/ChangeLog
https://invisible-island.net/ncurses/NEWS.html#t20181006

I retested with clang, icc, gcc on Fedora-15, -20 (to verify that my
earlier workaround was not needed) as well as Debian 8, Debian/testing.
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Thomas Dickey
2018-10-07 10:55:23 UTC
Permalink
Post by Thomas Dickey
I am sorry. Something weird happened when I was sending to you an email, my
email client had unexpectedly quit and I got confused whether the email was
sent; I resent you the other email where briefly told you what's been going
on. There a link has been, take a look
I saw it.
As I understood it would be nice if you regenerate configure with a fresh
version of 'autoconf'
that would be unlikely to solve this problem: at the time, I compared against
"fresh" autoconf, and its stub code did not fix the configuration that I was
interested in, nor did it improve a few related problems with weak symbols
on other platforms.
I'll take both into account (thanks)
I took that into account, but was reminded in the process of misinformation
provided by the developers on that topic.

So the only useful part of the bug report was the set of compiler options which
were used. I noticed some further issues on that, outside this bug report,
and made a to-do item to investigate - will probably not refer to Gentoo bug
reports for some time unless they provide _new_ information.
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Denis Pronin
2018-10-07 11:06:43 UTC
Permalink
Thomas, thank you for support
How could we obtain a new version of ncurses?
Post by Thomas Dickey
 > I am sorry. Something weird happened when I was sending to you an email, my
 > email client had unexpectedly quit and I got confused whether the email was
 > sent; I resent you the other email where briefly told you what's been going
 > on. There a link has been, take a look
 I saw it.
 > As I understood it would be nice if you regenerate configure with a fresh
 > version of 'autoconf'
 that would be unlikely to solve this problem: at the time, I compared against
 "fresh" autoconf, and its stub code did not fix the configuration that I was
 interested in, nor did it improve a few related problems with weak symbols
 on other platforms.
 I'll take both into account (thanks)
I took that into account, but was reminded in the process of misinformation
provided by the developers on that topic.
So the only useful part of the bug report was the set of compiler options which
were used. I noticed some further issues on that, outside this bug report,
and made a to-do item to investigate - will probably not refer to Gentoo bug
reports for some time unless they provide _new_ information.
--
https://invisible-island.net
ftp://ftp.invisible-island.net
Thomas Dickey
2018-10-07 15:23:46 UTC
Permalink
Post by Denis Pronin
Thomas, thank you for support
How could we obtain a new version of ncurses?
The sources are listed in (mostly-)weekly patch announcements.

For example

http://lists.gnu.org/archive/html/bug-ncurses/2018-10/msg00010.html

For ad hoc fetches

https://invisible-island.net/datafiles/current/ncurses.tar.gz

Getting the ebuild, etc., updated in Gentoo is something for the package
maintainer to answer. I recall some discussion in Gentoo's bug-reports
which would illustrate, but see that's been closed. If I see something
recent, I may add it to the FAQ.
--
Thomas E. Dickey <***@invisible-island.net>
https://invisible-island.net
ftp://ftp.invisible-island.net
Loading...