Secure service to service communication using Istio

We will use Istio to authentication and authorize services during communication using mutualTLS.

Prerequisites :

• IBM Cloud Account

• Kubernetes Cluster

• Istio CTL

• Github Repo Cloned

• Access to the cluster

One of the main challenges of managing microservices based solutions is how to properly secure not just the microservices themselves but also the communication between them. We will use Istio that can help us secure this communication without making changes to the application code itself.

We will first need to enable Istio for our cluster running on IBM Cloud. IBM provides managed Istio which allows seamless installation of Istio with other benefits such as automatic updates and lifecycle management of different components.

To enable Istio, navigate to your cluster and click on the Add-ons tab.

You will then see a tile for Managed Istio. Click on install to install istio in your cluster.

Wait for the installation to be ready. You will see Add on ready status as shown below

Once the add on is ready, Setup default configuration profile for Istio by running the following command in your terminal

istioctl manifest apply

Authentication

Istio offers mutual TLS as a full stack solution for transport authentication, which can be enabled without requiring service code changes. This solution:

• Provides each service with a strong identity representing its role to enable interoperability across clusters and clouds.

• Secures service-to-service communication.

• Provides a key management system to automate key and certificate generation, distribution, and rotation.

We will now create few services in our cluster and see how authentication works when we enable Istio.

To see how authentication works we will create 3 namespaces (foo, bar and legacy) with different services (sleep and httpbin) running as shown below. Foo and Bar namespaces are running services with an Envoy proxy to add istio capabilities while legacy namespace is running a service without an istio capabilities.

Step 1: Create namespaces

Create namespace foo, bar and legacy

kubectl create ns foo

kubectl create ns bar

kubectl create ns legacy

Step 2: Create services

Before running the following commands, make sure to change directory to istio-workflow in the cloned github repository

We will create the httpbin service in the foo

kubectl apply -f <(istioctl kube-inject -f httpbin.yaml) -n foo

Next let's create sleep service in the bar (with istio) and legacy (without istio) namespace.

kubectl apply -f <(istioctl kube-inject -f sleep.yaml) -n bar

kubectl apply -f sleep.yaml -n legacy

Make sure that the services are running in each namespace as shown below.

kubectl get pods -n foo

kubectl get pods -n bar

kubectl get pods -n legacy

The output should be as follows

$ kubectl get pods -n foo 
NAME                       READY   STATUS    RESTARTS   AGE
httpbin-55dcc67d67-dppwc   2/2     Running   0          6d7h

$ kubectl get pods -n bar
NAME                    READY   STATUS    RESTARTS   AGE
sleep-565f8679c-jzgxl   2/2     Running   0          6d7h

$ kubectl get pods -n legacy
NAME                    READY   STATUS    RESTARTS   AGE
sleep-f8cbf5b76-pxmg2   1/1     Running   0          6d7h

Step 3: Send request from bar to foo

By default (from version 1.4 onwards), Istio tracks the server workloads migrated to Istio proxies, and configures client proxies to send mutual TLS traffic to those workloads automatically, and to send plain text traffic to workloads without sidecars.

Thus, all traffic between workloads with proxies uses mutual TLS, without you doing anything. For example, take the response from a request to httpbin/header. When using mutual TLS, the proxy injects the X-Forwarded-Client-Cert header to the upstream request to the backend. That header’s presence is evidence that mutual TLS is used.

Replace <pod-name-in-bar> with the name of the pod running in bar namespace.

kubectl exec <pod-name-in-bar> -n bar -- curl http://httpbin.foo:8000/headers 

The output will be as follows and you can see the X-Forwarded-Client-Cert towards the end.

"headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.foo:8000", 
    "User-Agent": "curl/7.64.0", 
    "X-B3-Parentspanid": "833746fa90d48a4d", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "26765f211c8f2fb5", 
    "X-B3-Traceid": "2dbf96936064d5ca833746fa90d48a4d", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/foo/sa/httpbin;Hash=7b316699171742e559d139e115f2c19fff6708fb5ae2011a56c7333615396c66;Subject=\"\";URI=spiffe://cluster.local/ns/bar/sa/sleep"
  }

Step 4: Send request from legacy to foo

Let's now send a request from legacy to foo and we will see how the X-Forwarded-Client-Cert header will be excluded.

Replace <pod-name-in-bar> with the name of the pod running in legacy namespace.

kubectl exec <pod-name-in-bar> -n legacy -- curl http://httpbin.foo:8000/headers 

The output will be as follows

  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.foo:8000", 
    "User-Agent": "curl/7.64.0", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "df4454d90cecf080", 
    "X-B3-Traceid": "a0f060dbb32ae043df4454d90cecf080"
  }

Step 5: Enabling Istio mutual TLS in STRICT mode

While Istio automatically upgrades all traffic between the proxies and the workloads to mutual TLS between, workloads can still receive plain text traffic. To prevent non-mutual TLS for the whole mesh, set a mesh-wide peer authentication policy to set mutual TLS mode to STRICT.

kubectl apply -f policies-istio/authentication-strict.yaml

Once the authentication policy is enabled, request from legacy to foo should not be allowed.

kubectl exec <pod-name-in-bar> -n legacy -- curl http://httpbin.foo:8000/headers 

Output will be as follows

curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

Authorization

Each Envoy proxy runs an authorization engine that authorizes requests at runtime. When a request comes to the proxy, the authorization engine evaluates the request context against the current authorization policies, and returns the authorization result, either ALLOW or DENY. Operators specify Istio authorization policies using .yaml files.

Step 1: Setup sample application

The sample application shows information about books. The services communicate as shown below:

We will first set up a sample application to understand how authorization works

kubectl apply -f <(istioctl kube-inject -f bookinfo.yaml)

Confirm all services and pods are correctly defined and running:

kubectl get pods

kubectl get services

Now that the Bookinfo services are up and running, you need to make the application accessible from outside of your Kubernetes cluster, e.g., from a browser. An Istio Gateway is used for this purpose.

kubectl apply -f bookinfo-gateway.yaml

Confirm the gateway has been created:

kubectl get gateway

Follow the following to set the INGRESS_HOST and INGRESS_PORT variables for accessing the gateway.

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')

Set GATEWAY_URL:

export GATEWAY_URL=$INGRESS_HOST:$INGRESS_PORT

You can now point your browser to http://$GATEWAY_URL/productpage to view the Bookinfo web page. You will see the page properly loaded with details and reviews of the book.

Step 2: Deny all traffic between services.

We will now apply a policy to deny all traffic.

kubectl apply -f policies-istio/authorization-deny-all.yaml

Reload http://$GATEWAY_URL/productpage and you not be able to access the page anymore (it might take few seconds for the policy to take effect).

Step 3: Allow access to product service.

Run the following command to create a productpage-viewer policy to allow access withGET method to the productpage workload.

kubectl apply -f policies-istio/authorization-product.yaml

Reload http://$GATEWAY_URL/productpage and you will be able to access the product service without access to details and reviews (it might take few seconds for the policy to take effect).

Step 4: Allow product service to access details service

Run the following command to create the details-viewer policy to allow the productpage workload, which issues requests using the cluster.local/ns/default/sa/bookinfo-productpage service account, to access the details workload through GET methods:

kubectl apply -f policies-istio/authorization-details.yaml

Reload http://$GATEWAY_URL/productpage and you will be able to access the product service with access to details (it might take few seconds for the policy to take effect).

You can do the same for reviews to be accessed by product service.