Failed to list *v1alpha1.Order: orders.certmanager.k8s.io is forbidden

Question

I configured automated SSL certificate management few months ago as described here: http://docs.cert-manager.io/en/latest/tutorials/acme/dns-validation.html for domains: <myhost>.com and dev.<myhost>.com. So I have two namespaces: prod for <myhost>.com and dev for dev.<myhost>.com. In each namespace I have ingress controller and Certificate resource to store certificate to secret. It's working fine and ClusterIssuer automatically updates certificates.

But few days ago I tried to add new domain: test.<myhost>.com in test namespace with absolutely same configuration of ingress and certificate as in prod or dev namespace (expect host name and namespace):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    kubernetes.io/tls-acme: 'true'
  name: app-ingress
  namespace: test
spec:
  tls:
  - hosts:
    - test.<myhost>.com
    secretName: letsencrypt-tls
  rules:
    - host: test.<myhost>.com
      http:
        paths:
        - backend:
            serviceName: web
            servicePort: 80
          path: /
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: cert-letsencrypt
  namespace: test
spec:
  secretName: letsencrypt-tls
  issuerRef:
    name: letsencrypt-prod-dns
    kind: ClusterIssuer
  commonName: 'test.<myhost>.com'
  dnsNames:
  - test.<myhost>.com
  acme:
    config:
    - dns01:
        provider: dns
      domains:
      - test.<myhost>.com

and this configuration doesnt work: certificate can't be found in secret, ingress is using "app-ingress-fake-certificate".

cert-manager pod shows a lot of similar errors:

pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Challenge: challenges.certmanager.k8s.io is forbidden: User "system:serviceaccount:kube-system:cert-manager" cannot list challenges.certmanager.k8s.io at the cluster scope
pkg/client/informers/externalversions/factory.go:72: Failed to list *v1alpha1.Order: orders.certmanager.k8s.io is forbidden: User "system:serviceaccount:kube-system:cert-manager" cannot list orders.certmanager.k8s.io at the cluster scope

and certificate is not trying to get certificate (kubectl describe -ntest cert-letsencrypt):

API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata: ...
Spec:
  Acme:
    Config:
      Dns 01:
        Provider:  dns
      Domains:
        test.<myhost>.com
  Common Name:  test.<myhost>.com
  Dns Names:
    test.<myhost>.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod-dns
  Secret Name:  letsencrypt-tls
Events:         <none>

It should have any status as certificates on other namespaces.

I can't understand why this configuration worked before but can't work now.

I'm not sure it's related, but I updated kubernetes using kops few weeks ago, current version is:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"archive", BuildDate:"2018-10-12T16:56:06Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"a21fdbd78dde8f5447f5f6c331f7eb6f80bd684e", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Answer 1

The cause of this issue was Kubernetes upgrade from 1.9 to 1.10. To fix it you need to upgrade cert-manager to 0.5.x version.

It may be not possible to upgrade from 0.4.x to 0.5.x using helm because of bug https://github.com/jetstack/cert-manager/issues/1134 in such case you need to store all issuers and certificates configurations then delete cert-manager 0.4.x and install 0.5.x, then apply all issuers and certificates configurations from first step.