bpo-40479: Fix hashlib issue with OpenSSL 3.0.0

OpenSSL 3.0.0-alpha2 was released today. The FIPS_mode() function has
been deprecated and removed. It no longer makes sense with the new
provider and context system in OpenSSL 3.0.0.

EVP_default_properties_is_fips_enabled() is good enough for our needs in
unit tests. It's an internal API, too.

Signed-off-by: Christian Heimes <[email protected]>

Author: tiran

Misc/NEWS.d/next/Library/2020-05-15-17-38-21.bpo-40479.yamSCh.rst

@@ -0,0 +1 @@
+The :mod:`hashlib` now compiles with OpenSSL 3.0.0-alpha2.

Modules/_hashopenssl.c

@@ -1109,19 +1109,25 @@ _hashlib.get_fips_mode -> int
 
 Determine the OpenSSL FIPS mode of operation.
 
+For OpenSSL 3.0.0 and newer it returns the state of the default provider
+in the default OSSL context. It's not quite the same as FIPS_mode() but good
+enough for unittests.
+
 Effectively any non-zero return value indicates FIPS mode;
 values other than 1 may have additional significance.
-
-See OpenSSL documentation for the FIPS_mode() function for details.
 [clinic start generated code]*/
 
 static int
 _hashlib_get_fips_mode_impl(PyObject *module)
-/*[clinic end generated code: output=87eece1bab4d3fa9 input=c2799c3132a36d6c]*/
+/*[clinic end generated code: output=87eece1bab4d3fa9 input=2db61538c41c6fef]*/
 
 {
+    int result;
+#if OPENSSL_VERSION_NUMBER >= 0x30000000L
+    result = EVP_default_properties_is_fips_enabled(NULL);
+#else
     ERR_clear_error();
-    int result = FIPS_mode();
+    result = FIPS_mode();
     if (result == 0) {
         // "If the library was built without support of the FIPS Object Module,
         // then the function will return 0 with an error code of
@@ -1134,6 +1140,7 @@ _hashlib_get_fips_mode_impl(PyObject *module)
         }
     }
     return result;
+#endif
 }
 #endif  // !LIBRESSL_VERSION_NUMBER