Running kubernetes autoscalar

I have a replication controller running with the following spec:

apiVersion: v1
kind: ReplicationController
metadata:
  name: owncloud-controller
spec:
  replicas: 1
  selector:
    app: owncloud
  template:
    metadata:
      labels:
        app: owncloud
    spec:
      containers:
      - name: owncloud
        image: adimania/owncloud9-centos7
        ports:
          - containerPort: 80
        volumeMounts:
          - name: userdata
            mountPath: /var/www/html/owncloud/data
        resources:
          requests:
            cpu: 400m
      volumes:
        - name: userdata
          hostPath:
            path: /opt/data

Now I run a hpa using autoscale command.

$ kubectl autoscale rc owncloud-controller --max=5 --cpu-percent=10

I have also started heapster using kubernetes run command.

$ kubectl run heapster --image=gcr.io/google_containers/heapster:v1.0.2 --command -- /heapster --source=kubernetes:http://192.168.0.103:8080?inClusterConfig=false --sink=log

After all this, the autoscaling never kicks in. From logs, it seems that the actual CPU utilization is not getting reported.

$ kubectl describe hpa owncloud-controller
Name:               owncloud-controller
Namespace:          default
Labels:             <none>
Annotations:            <none>
CreationTimestamp:      Thu, 26 May 2016 14:24:51 +0530
Reference:          ReplicationController/owncloud-controller/scale
Target CPU utilization:     10%
Current CPU utilization:    <unset>
Min replicas:           1
Max replicas:           5
ReplicationController pods: 1 current / 1 desired
Events:
  FirstSeen LastSeen    Count   From                SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  44m       8s      92  {horizontal-pod-autoscaler }            Warning     FailedGetMetrics    failed to get CPU consumption and request: metrics obtained for 0/1 of pods
  44m       8s      92  {horizontal-pod-autoscaler }            Warning     FailedComputeReplicas   failed to get CPU utilization: failed to get CPU consumption and request: metrics obtained for 0/1 of pods

What am I missing here?

Most probably heapster is running in a wrong namespace ("default"). HPA expects heapster to be in "kube-system" namespace. Please, add --namespace=kube-system to kubectl run heapster command.

I installed hepaster under the name space "kube-system" and it worked. After running heapster, make sure it's running before you use HPA for your application.

How to run Heapster with Kubernetes cluster

I put all files here https://gitlab.com/abushoeb/kubernetes/tree/master/heapster. They are collected from the official Kubernetes Repository and made minor changes.

How to run Heapster

Go to the directory heapster where you have grafana.yaml, heapster.yaml and influxdb.yaml and run following command $ kubectl create -f .

How to stop Heapster

Go to the same heapster directory and then run following command $ kubectl delete -f .

How to check Heapster is running

You can access hepaster metric model from the pod where heapster is running to make sure heapster is working. It can be accessed via web browser by accessing http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/. The same result can be seen by executing following command. $ curl -L http://heapster-pod-ip:heapster-service-port/api/v1/model/metrics/

If you see the list of metrics then hepaster is running correctly. You can also browse grafana dashboard to see it (find the ip of the pod where grafana is running and the access it http://grafana-pod-ip:grafana-service-port).

Full documentation of Heapster Metric Model are avaibale here.

Also just run ($ kubectl cluster-info) and see if it shows results like this:

Kubernetes master is running at https://cluster-ip:6443

Heapster is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/heapster

kubernetes-dashboard is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

monitoring-grafana is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-grafana

monitoring-influxdb is running at https://cluster-ip:6443/api/v1/proxy/namespaces/kube-system/services/monitoring-influxdb

Check influxdb

You can also check influxdb if it has data in it. Install Influxdb Client on your local machine to get connected to infuxdb database. $ influx -host <cluster-ip> -port <influxdb-service-port>

Some Sample influxdb queries