As I don\'t know the openssl command\'s arguments by heart, I am used to referring to the same SO answer whenever I need to create self-signed certificates (for testing environm
openssl req
creates a CSR or a CA root certificate. See the man page. It is not what you want. The second set of steps is correct.
Short way (e.g. with OpenSSL 1.1.0f and Apache 2.4.37):
openssl genrsa -out notEncodedPk.key 3072
openssl req -new -out website.csr -sha256 -key notEncodedPk.key
openssl x509 -req -in website.csr -days 365 -signkey notEncodedPk.key -out website.cert -outform PEM
genrsa generates a 3072 bit RSA-Key. (The system should be online for some time to have good data in /dev/(u)random for seeding.) There is no need to generate an encrypted PK (1) and then use rsa to remove the password afterwards. (Maybe earlier versions of the tools required a password?)
req creates the certificate signing request and uses the PK for the signature. Providing something like -sha256 for the digest is optional. (3) Provide your infos in the interactive questionare. Ensure to put your site domain in "Common name:", otherwise the Apache will throw a warning (AH01909) and browsers will throw an "invalid certificate" message because the URL/domain does not match the certificate data (2). Leave "A challange password:" empty.
Use x509 to create a self-signed certificate with -signkey (the subject is copied to issuer). Normally the command works on certificates but with -req it accepts a CSR as an input. Then use your PK for signing the certificate. (-outform and -days are optional, with 30 days as the default value for the latter.)
Problem source:
As user207421 already stated: req creates a CSR OR it creates a self-signed root-CA-like certificate, therefore the typical tutorial tip
openssl req -x509 -nodes -days 365 -newkey rsa:3072 -sha256 -keyout website.key -out website.cert
is short but normally not what you want. You can also compare created certificates with
openssl x509 -text -noout -in website.cert
In the certificate, created with the single-line command, you see a section "X509v3 extensions:" with "X509v3 Basic Constraints: critical CA:TRUE". This is exactly the Apache warning message.
Instead, if you create the certificate with the three steps, the "X509v3 extensions:" section is not included into the certificate.
Appendix:
(1) Securing the PK with a password is a good idea in most cases. If the PK is stored without encryption, make sure to restrict access to root. If you use a password, you have to use the -passout/-passin options, but be aware that a simple "x" does not work anymore because some OpenSSL tools require at least 4 characters (otherwise: "result too small/bad password read"). Additionally in Apache you have to use something like SSLPassPhraseDialog buildin to manually enter the required password for the PK (or even for all PKs/certs) during Apache startup.
(2) Anyway, browsers will display a warning for self-signed certificates.
(3) Using SHA-1 would be inadequate for such a large RSA-key. In general, it is a good idea to review your openssl.conf, e.g. in Debian 9 in /etc/ssl/openssl.conf, which contains various defaults, for example signer_digest = sha256.
In the Debian 9 file, you also find in the [req] section a line x509_extensions=v3_ca and this is the reason, why the req command in combination with the -x509 option adds the CA-related extension (basicContraints=critical,CA:true), if used in the single-line style to create a self-signed certificate.
Addidionally you might notice a comment-line # req_extensions=v3_req. Because this line is commented out (in Debian 9 default openssl.cnf), the simple usage of the req command does not include any extensions.
Note that you might use this line in a modified file to add Subject Alternative Name's to the certificate, e.g. so it can handle multiple (sub-)domains (normally a much better choice than using e wildcard in CN, e.g. *.example.com).
I had the same problem just today on Debian 9 stretch and I tried your solution to generate a new certificate using your method and it did not work. The warning in Apache was exactly the same.
I found out that the problem was that in my browser were stored other 6 certificates with the same FQDN. I erased the certificates and the problem has gone.
EDIT: Well, there's still the warning actually but at least everything is working.