25 Aug 2021

Using .NET with OpenSSL in CentOS Stream 9

Or, how to deal with error:0E07607: configuration file routines….

TLDR:

# Only for .NET 5 or .NET Core 3.1
sudo dnf install -y compat-openssl11
export OPENSSL_CONF=/etc/pki/openssl11.cnf
dotnet whatever

Introduction

How do you get .NET applications - whether the SDK/runtime itself or a self-contained application - working on RHEL 9?

Generally, this should be a walk in the park. .NET is a cross platform runtime. It explicitly supports RHEL. There shouldn’t be any surprises here.

Unfortunately, RHEL 9 (or CentOS Stream 9, rather) is still in development. It has also switched to OpenSSL 3.0, which currently-released versions of .NET don’t support. Even forcing OpenSSL 1.1 leads to an obscure error:0E07607: configuration file routines error.

In the rest of this post, I will walk you through running a simple .NET Core network-using application on CentOS Stream 9. I will be using container images, but only because it makes things easier to reproduce. There’s nothing here that’s specific to container images.

CentOS Stream? Why not RHEL 9?

CentOS Stream 9 is the in-development version of what will eventually become RHEL 9. You can find more details about how RHEL 9 and CentOS Stream 9 relate here . I am going to use CentOS Stream 9 in the rest of this post.

First, let’s get our hands on a container we can use for trying things out. I am going to use the latest CentOS Stream 9 image: quay.io/centos/centos:stream9-development. Other CentOS images are available too . If you are wondering, these images are official images created by the CentOS Stream folks .

I will use podman through this post. Feel free to replace the podman command with docker if you want. The podman and docker commands accept the same set of sub-commands and arguments. There shouldn’t be a function difference b

This image claims it’s CentOS Stream 9, and even tells us that it should reflect the next release of RHEL 9:

$ podman run -it quay.io/centos/centos:stream9-development cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Install and Use .NET Core

Let’s start with a simple Dockerfile. We will use this to write and run a trivial .NET Core 3.1 application:

FROM quay.io/centos/centos:stream9-development

RUN dnf install -yq wget libicu openssl

RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh && \
    bash dotnet-install.sh && \
    export PATH=$PATH:$HOME/.dotnet/ && \
    dotnet new console -o /app

Here’s what this Dockerfile does, line by line.

Use the CentOS Stream 9 container image
Install some dependencies
- wget is needed to download .NET Core itself
- libicu and openssl are required .NET Core dependencies
Get the .NET Core installation script (dotnet-install.sh)
Use the installation script to install .NET Core 3.1, which installs it to ~/.dotnet (the same as $HOME/.dotnet)
Update $PATH to include the new .NET installation directory, to make future invocations of dotnet work.
Create a new Hello-World application at /app.

Now, let’s build this Dockerfile:

$ podman build --tag centos-dotnet-container --file Dockerfile
STEP 1: FROM quay.io/centos/centos:stream9-development
Trying to pull quay.io/centos/centos:stream9-development...
Getting image source signatures
Copying blob 21c097e6cb77 done
Copying config 236076175b done
Writing manifest to image destination
Storing signatures
STEP 2: RUN dnf install -yq wget libicu openssl

Installed:
  libicu-67.1-9.el9.x86_64         openssl-1:3.0.0-0.beta2.6.el9.x86_64
  wget-1.21.1-6.el9.x86_64

--> e6a20071b75
STEP 3: RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh &&     bash dotnet-install.sh &&     export PATH=$PATH:$HOME/.dotnet/ &&     dotnet new console -o /app
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
... SNIP ...
Write your first app: https://aka.ms/first-net-core-app
--------------------------------------------------------------------------------------
No usable version of libssl was found
container exited on segmentation fault
Error: error building at STEP "RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh &&     bash dotnet-install.sh &&     export PATH=$PATH:$HOME/.dotnet/ &&     dotnet new console -o /app": error while running runtime: exit status 1

As you can see, this fails to work with a No usable version of libssl was found. libssl is a part of OpenSSL. This message means that .NET couldn’t find a version of OpenSSL that it could use.

Switch to OpenSSL 1.1

If you check the container, it has OpenSSL 3.0 already installed:

$ podman run -it quay.io/centos/centos:stream9-development rpm -q openssl-libs
openssl-libs-3.0.0-0.beta2.6.el9.x86_64

Does that mean .NET doesn’t know how to make use of that?

Yes. It turns out .NET Core 3.1 and .NET 5.0 only know how to make use of OpenSSL 1.0 and 1.1. Support for OpenSSL 3.0 will be included in .NET 6 and later.

So let’s install OpenSSL 1.1, which .NET Core 3.1 can use.

The CentOS Stream 9 package for OpenSSL 1.1 is called compat-openssl11. Due to a bug, it’s not currently available in the CentOS Stream 9 package repositories. It should be available soon. For now, we will work around that by manually getting the package and installing it.

FROM quay.io/centos/centos:stream9-development

RUN dnf install -yq wget libicu openssl

# This is the correct method once the issue is fixed: RUN dnf install -y compat-openssl11
# This next line is the workaround:
RUN wget -q --no-check-certificate https://kojihub.stream.rdu2.redhat.com/kojifiles/packages/compat-openssl11/1.1.1k/2.el9/x86_64/compat-openssl11-1.1.1k-2.el9.x86_64.rpm && dnf install -yq ./compat-openssl11*.rpm

RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh && \
    bash dotnet-install.sh && \
    export PATH=$PATH:$HOME/.dotnet/ && \
    dotnet new console -o /app && \
    cd /app && \
    dotnet run

Running this should get us a working hello world:

$ podman build --tag centos-dotnet-container --file Dockerfile
STEP 1: FROM quay.io/centos/centos:stream9-development
STEP 2: RUN dnf install -yq wget libicu openssl
STEP 3: RUN wget -q --no-check-certificate https://kojihub.stream.rdu2.redhat.com/kojifiles/packages/compat-openssl11/1.1.1k/2.el9/x86_64/compat-openssl11-1.1.1k-2.el9.x86_64.rpm && dnf install -yq ./compat-openssl11*.rpm
STEP 4: RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh &&     bash dotnet-install.sh &&     export PATH=$PATH:$HOME/.dotnet/ &&     dotnet new console -o /app &&     cd /app &&     dotnet run
... SNIP ...
Getting ready...
The template "Console Application" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /app/app.csproj...
  Determining projects to restore...
  Restored /app/app.csproj (in 119 ms).

Restore succeeded.

Hello World!
STEP 5: COMMIT centos-dotnet-container
Successfully tagged localhost/centos-dotnet-container:latest
1f2ce75a289fd914567952c3ff209dcd8671382af9573922d03f464aa90d91cf

Is OpenSSL 1.1 Actually Working?

The error messages about libssl not being found have gone away. But we haven’t actually used any networking or cryptographic features. At this point, we don’t actually know if OpenSSL support in .NET is functional.

We can confirm, at least to certain extent, that .NET is working with OpenSSL with this trivial program at Program.cs.

using System;
using System.Net.Http;
using System.Threading.Tasks;

class Program
{
    static readonly HttpClient client = new HttpClient();

    static async Task Main()
    {
        try
        {
            string response = await client.GetStringAsync("http://dot.net/");
            Console.WriteLine($"Got {response.Length} bytes");
        }
        catch (HttpRequestException e)
        {
            Console.Error.WriteLine(e.ToString());
        }
    }
}

This program is based on the HttpClient sample . It simply connects to https://dot.net/ and gets the contents of returned by that URL as a string.

Since the URL is an https resource, it ends up using a large part of the cryptographic stack of .NET, much of which is using the underlying OpenSSL library on Linux. In other words, if we can access this URL, we can safely assume that a large number of cryptographic features in OpenSSL can be correctly used by our .NET runtime.

Let’s add this into our application through our Dockerfile.

FROM quay.io/centos/centos:stream9-development

RUN dnf install -yq wget libicu openssl

# This is the correct method once the issue is fixed: RUN dnf install -y compat-openssl11
# This next line is the workaround:
RUN wget -q --no-check-certificate https://kojihub.stream.rdu2.redhat.com/kojifiles/packages/compat-openssl11/1.1.1k/2.el9/x86_64/compat-openssl11-1.1.1k-2.el9.x86_64.rpm && dnf install -yq ./compat-openssl11*.rpm

RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh && \
    bash dotnet-install.sh && \
    export PATH=$PATH:$HOME/.dotnet/ && \
    dotnet new console -o /app

ADD Program.cs /app/

CMD cd /app && \
    export PATH=$PATH:$HOME/.dotnet/ && \
    dotnet run

Then, we can build it.

$ podman build --tag centos-dotnet-container --file Dockerfile
... SNIP ...
STEP 7: COMMIT centos-dotnet-container
--> e6f6459a92e
Successfully tagged localhost/centos-dotnet-container:latest
e6f6459a92eb673c4d869790584d27ecdbb84d085aff562df3b9271d81f9ffd4

Finally, we run our application:

$ podman run -it centos-dotnet-container
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.TypeInitializationException: The type initializer for 'SslMethods' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'Ssl' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'SslInitializer' threw an exception.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0E076071:configuration file routines:module_run:unknown module name
   at Interop.SslInitializer..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl..cctor()
 ... SNIP ...
   at System.Net.Http.HttpClient.GetStringAsyncCore(Task`1 getTask)
   at Program.Main() in /app/Program.cs:line 13

Well, that’s certainly not an error I had ever seen before.

A quick google for 0E076071 brings us to https://github.com/dotnet/runtime/issues/27792 . My take away from that discussion is that an unknown module name error basically means that the current version of OpenSSL can not parse the OpenSSL configuration file (located at /etc/pki/tls/openssl.cnf for CentOS Stream 9). We need to use a different OpenSSL version, or use a different configuration file.

Following the advice from https://bugzilla.redhat.com/show_bug.cgi?id=1694850 , we should override the default and unusable configuration with a configuration usable by OpenSSL 1.1.

If we look for all .cnf files owned/installed by the compat-openssl11 package, we can find where a usable configuration file is:

$ podman run -it centos-dotnet-container rpm -ql compat-openssl11 | grep cnf
/etc/pki/openssl11.cnf

Then we can modify our Dockerfile to export OPENSSL_CONF=/etc/pki/openssl11.cnf to override the default OpenSSL 1.1 configuration:

FROM quay.io/centos/centos:stream9-development

RUN dnf install -yq wget libicu openssl

# This is the correct method once the issue is fixed: RUN dnf install -y compat-openssl11
# This next line is the workaround:
RUN wget -q --no-check-certificate https://kojihub.stream.rdu2.redhat.com/kojifiles/packages/compat-openssl11/1.1.1k/2.el9/x86_64/compat-openssl11-1.1.1k-2.el9.x86_64.rpm && dnf install -yq ./compat-openssl11*.rpm

RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh && \
    bash dotnet-install.sh && \
    export PATH=$PATH:$HOME/.dotnet/ && \
    dotnet new console -o /app

ADD Program.cs /app/

CMD cd /app && \
    export PATH=$PATH:$HOME/.dotnet/ && \
    export OPENSSL_CONF=/etc/pki/openssl11.cnf && \
    dotnet run

Now, let’s build and run it one last time.

$ podman build --tag centos-dotnet-container --file Dockerfile
STEP 1: FROM quay.io/centos/centos:stream9-development
STEP 2: RUN dnf install -yq wget libicu openssl
STEP 3: RUN wget -q --no-check-certificate https://kojihub.stream.rdu2.redhat.com/kojifiles/packages/compat-openssl11/1.1.1k/2.el9/x86_64/compat-openssl11-1.1.1k-2.el9.x86_64.rpm && dnf install -yq ./compat-openssl11*.rpm
STEP 4: RUN wget -q --no-check-certificate https://dot.net/v1/dotnet-install.sh &&     bash dotnet-install.sh &&     export PATH=$PATH:$HOME/.dotnet/ &&     dotnet new console -o /app
STEP 5: ADD Program.cs /app/
STEP 6: CMD cd /app &&     export PATH=$PATH:$HOME/.dotnet/ &&     export OPENSSL_CONF=/etc/pki/openssl11.cnf &&     dotnet run
STEP 7: COMMIT centos-dotnet-container
$ podman run -it centos-dotnet-container
Got 61174 bytes

Finally, it all works!

What’s next

To summarize, running .NET Core 3.1 (and .NET 5) on CentOS Stream 9 involves 2 main steps:

Installing OpenSSL 1.1
Forcing .NET to use OpenSSL 1.1’s configuration file instead of the default OpenSSL configuration

Now that you know how to work around things, you can figure out how to get older .NET applications running on RHEL 9, including standalone applications.

.NET 6, which is on the horizon, will support OpenSSL 3.0 natively and will not need any of the workarounds we had to resort to.

That’s just one of the reasons to get excited about .NET 6. Download .NET 6 today and try it out on your favourite Linux distribution!

And of course, like previous versions of .NET, .NET 6 will be available in CentOS Stream 9 as dotnet-sdk-6.0 at some point in the future.