Compiling OpenSSL in Solaris 10 with GCC 5.1.0

I got this error when trying to compile OpenSSL 1.0.2d on my Solaris machines:

[...]
gcc -I.. -I../.. -I../modes -I../asn1 -I../evp -I../../include  -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -m64 -O3 -Wall -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM   -c -o randfile.o randfile.c
In file included from /usr/include/stdio.h:21:0,
                 from randfile.c:65:
/usr/local/lib/gcc/i386-pc-solaris2.10/5.1.0/include-fixed/sys/feature_tests.h:346:2: error: #error "Compiler or options invalid for pre-UNIX 03 X/Open applications     and pre-2001 POSIX applications"
 #error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
  ^
<builtin>: recipe for target 'randfile.o' failed
make[2]: *** [randfile.o] Error 1
make[2]: Leaving directory '/tmp/z/openssl-1.0.2d/crypto/rand'
Makefile:88: recipe for target 'subdirs' failed
make[1]: *** [subdirs] Error 1
make[1]: Leaving directory '/tmp/z/openssl-1.0.2d/crypto'
Makefile:283: recipe for target 'build_crypto' failed
make: *** [build_crypto] Error 1

When I get an error like this my next step is to try to compile again the already (own) compiled version present in my system. In this particular case I downloaded and recompiled OpenSSL 1.0.2c again. It fails with the same error, but it was correctly compiled a month ago. Given this, I know that the problem is not with the changes in the new release but changes in my environment.

Notably I upgraded my installed GCC version from 4.9.2 to 5.1.0 a few days ago.

The source code raising the error didn't changed between GCC 4.9.2 and 5.1.0:

$ diff -u \
    /usr/local/lib/gcc/i386-pc-solaris2.10/4.9.2/include-fixed/sys/feature_tests.h  \
    /usr/local/lib/gcc/i386-pc-solaris2.10/5.1.0/include-fixed/sys/feature_tests.h \
$ echo $?
0

Let's see it:

/*
 * It is invalid to compile an XPG3, XPG4, XPG4v2, or XPG5 application
 * using c99.  The same is true for POSIX.1-1990, POSIX.2-1992, POSIX.1b,
 * and POSIX.1c applications. Likewise, it is invalid to compile an XPG6
 * or a POSIX.1-2001 application with anything other than a c99 or later
 * compiler.  Therefore, we force an error in both cases.
 */
#if defined(_STDC_C99) && (defined(__XOPEN_OR_POSIX) && !defined(_XPG6))
#error "Compiler or options invalid for pre-UNIX 03 X/Open applications \
        and pre-2001 POSIX applications"
#elif !defined(_STDC_C99) && \
        (defined(__XOPEN_OR_POSIX) && defined(_XPG6))
#error "Compiler or options invalid; UNIX 03 and POSIX.1-2001 applications \
        require the use of c99"
#endif

Quite interesting, indeed. My first suspicion was that GCC 5.1.0 is defining _STDC_C99, but I was mistaken:

$ gcc -dM -E - </dev/null|grep -i C99

Humm...

Let's take a look to the sourcecode raising the issue, openssl-1.0.2d/crypto/rand/randfile.c. I see this suspicious code:

59
60
61
62
 
 /* We need to define this to get macros like S_IFBLK and S_IFCHR */
 #if !defined(OPENSSL_SYS_VXWORKS)
 # define _XOPEN_SOURCE 500
 #endif

Apparently setting _XOPEN_SOURCE to 500 enables some additional flags in GCC 5.1.0. Interestingly this setting is enabled in every system beside VxWorks, but maybe we don't need it in Solaris.

Let's delete those lines and try to recompile.

Everything goes well this time. Solaris doesn't need that #define.

make check is green.

Success!.

Bonus Track

I was curious and I checked the usage of S_IFBLK and S_IFCHR in the code. I found this:

154
155
156
157
158
159
160
161
162
163
164
165
166
 
 #if defined(S_IFBLK) && defined(S_IFCHR) && !defined(OPENSSL_NO_POSIX_IO)
     if (sb.st_mode & (S_IFBLK | S_IFCHR)) {
         /*
          * this file is a device. we don't want read an infinite number of
          * bytes from a random device, nor do we want to use buffered I/O
          * because we will waste system entropy.
          */
         bytes = (bytes == -1) ? 2048 : bytes; /* ok, is 2048 enough? */
 # ifndef OPENSSL_NO_SETVBUF_IONBF
         setvbuf(in, NULL, _IONBF, 0); /* don't do buffered reads */
 # endif                         /* ndef OPENSSL_NO_SETVBUF_IONBF */
     }
 #endif

This is interesting. I would expect S_IFBLK and S_IFCHR detection at configuration time, but the programmer wrote a compilation-time check. That is, if those #define are missing, the code will compile anyway, skipping some checks.

So maybe Solaris needs #define _XOPEN_SOURCE 500 after all. I checked /usr/include/sys/stat.h header file to be sure and, fortunately, I confirm that those flags are always available, independently of _XOPEN_SOURCE value.

Just to be sure we could do this change:

--- crypto/rand/randfile.c.old  2015-07-10 02:44:30.171640289 +0200
+++ crypto/rand/randfile.c      2015-07-10 02:54:20.081824839 +0200
@@ -57,7 +57,7 @@
  */

 /* We need to define this to get macros like S_IFBLK and S_IFCHR */
-#if !defined(OPENSSL_SYS_VXWORKS)
+#if !defined(OPENSSL_SYS_VXWORKS) && !defined(__sun)
 # define _XOPEN_SOURCE 500
 #endif

@@ -82,6 +82,12 @@
 # include <fcntl.h>
 #endif

+#if defined(__sun)
+# if !defined(S_IFBLK) || !defined(S_IFCHR)
+#  error "We need S_IFBLK and S_IFCHR"
+# endif
+#endif
+
 #ifdef _WIN32
 # define stat    _stat
 # define chmod   _chmod

Better safe than sorry.