Is there anything worse than managing a cloud service’s credentials and connection strings to infrastructure services? Well, maybe strawberries, but that is beside the point.
Every secret you must manage has the potential for errors and is a security liability. Oh, and as we programmers love, it is also additional tedious work. Can you do something about that when you deploy your services to an Azure Kubernetes cluster? Yes, and it is called Azure Active Directory Workload Identity – catchy. It is the successor to Pod Managed Identity, which suffered a Google fate and did not make it past its preview status.
Note: At the time of writing, Azure AD Workload Identity is also still in preview. From what I could find, it is slated to reach the Generally Available status (GA) at the end of 2022.
Visit Microsoft’s documentation for all the gritty details. The short version is that it saves a lot of the headaches that come with an AD application registration and Service Principal.
There is no password to configure for the service that uses the identity.
There is no password to rotate after X amount of time.
There is no need to do anything in Azure AD, bypassing the one thing that IT Ops departments usually guard with their life. Besides the network. And the coffee maker.
A Managed Identity lives in your Resource Group along with other infrastructure resources and does not require a password. Azure manages that part for you. Yet still, you can assign RBAC permissions to it as you would to a Service Principal.
Enter Sandman Azure AD Workload Identity.
AKS can have a System- or User-Assigned Managed Identity. But all that does is allow AKS to access resources it requires for its operation, like a load balancer. It is not a thing you can take advantage of in your microservice that you have imprisoned inside a pod. Nor would you want to. Your microservice is not AKS, and it should not impersonate one.
Additionally, you would want every one of your microservices to have its own identity so you have fine-grained control over what other services your application can access. In the olden days, when usernames and passwords were more pervasive, it was essential to revoke only a specific set of credentials if they were ever compromised, thus limiting the impact on the system. Managed Identities cannot leak credentials, but your microservice could still get hacked, and this also limits the impact to a single Managed Identity and the resources it can access.
But the question was how to enable this for a pod, so let me finally answer that.
First, you must install the az command line utility and the aks-preview extension. If you have already done so, perform an update to be safe.
az extension add/update --name aks-preview
The minimum requirements are az version 2.40+, aks-preview 0.5.102+, and Kubernetes 1.22+.
Now you can create a new AKS cluster or update an existing one with the OIDC issuer feature. I will show a diagram of how all the pieces fit together once they are on the table. Let’s assume an upgrade of an existing testing cluster named mi-test-aks.
az aks update -g mi-test-group --name mi-test-aks --enable-oidc-issuer
With this feature enabled, you can retrieve the issuer URL with this command.
az aks show --resource-group mi-test-group --name mi-test-aks --query "oidcIssuerProfile.issuerUrl" -otsv
It spits out something like this.
https://oidc.prod-aks.azure.com/0eb05a4c-2d75-4568-ad04-0bb03f6a99b1/
The / at the end is very important. If you forget it in later steps, you do not pass Go, do not collect 200 money, and go directly to jail, where you must solve the riddle of unhelpful error messages like this for eternity.
Azure Identity => ERROR in getToken() call for scopes [https://vault.azure.net/.default]: Server returned HTTP response code: 400 for URL
The official documentation has a dedicated troubleshooting section for the / error that mentions yet another failure reason.
The next step is to install a mutating admission webhook that injects the identity information as environment variables into your pods. I have found two ways of enabling this functionality, but unfortunately, I failed to verify them in isolation. I had both enabled simultaneously, and once I had the AKS feature flag set, it could not be reverted. So, let me start with this option.
az aks update -g mi-test-group --name mi-test-aks --enable-workload-identity
Another way that is independent of AKS is an installation via Helm.
helm repo add azure-workload-identity https://azure.github.io/azure-workload-identity/charts
helm repo update
helm install workload-identity-webhook azure-workload-identity/workload-identity-webhook \
--namespace azure-workload-identity-system \
--create-namespace \
--set azureTenantID=a1cd6959-ac13-4052-987d-067f854223c3
The result is two additional controllers in the azure-workload-identity-system namespace.
Speculation: I think both approaches are interchangeable on AKS.
The documentation always talks about installing the admission webhook so I would opt for the Helm method.
You also require a Managed Identity, of course. For the rest of this tutorial, I use the client-id 0ea4d539-3998-43a6-b30d-317889dc34cd as an example. You can get it from the Azure Portal if you like the visual approach or with some command line fu.
(To tease my Linux-using colleagues, I went with PowerShell syntax 😁)
PS > $APPLICATION_NAME="managed-identity-test"
PS > $APPLICATION_CLIENT_ID="$(az ad sp list --display-name "${APPLICATION_NAME}" --query '[0].appId' -otsv)""
The AKS puzzle is now complete, and you can move on to deploying a microservice.
No, not really. Fortunately, you do not require much of it to achieve your goal. Azure AD Workload Identity attempts to utilize standard Kubernetes concepts, and one is the Service Account resource. You create one of those, link it to your deployment, link it to the Managed Identity, and you’re good to go.
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: 0ea4d539-3998-43a6-b30d-317889dc34cd
labels:
azure.workload.identity/use: "true"
name: managed-identity-test
namespace: platform
The two essential parts are the annotation and the label. The first is the link to the Managed Identity, and the second basically enables the feature. I assume that the admission webhook looks for it.
In the Deployment resource, you must set two things:
serviceAccountName.azure.workload.identity/use: "true".
apiVersion: apps/v1
kind: Deployment
metadata:
name: managed-identity-test
spec:
template:
metadata:
labels:
azure.workload.identity/use: "true"
spec:
serviceAccountName: managed-identity-test
...
More annotations are available on the Service Account and Deployment but are only necessary for specific situations.
What this all does is project several environment variables into the Pod containing authentication information, and I explain later how to utilize them in a Spring Boot microservice.
AZURE_CLIENT_ID: 0ea4d539-3998-43a6-b30d-317889dc34cd
AZURE_TENANT_ID: a1cd6959-ac13-4052-987d-067f854223c3
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
Lastly, you create a federated credential that binds the Service Account to the Managed Identity. It contains links to many resources you have created so far. The name of the Managed Identity and the Resource Group it resides in, the OIDC URL of AKS, and the Service Account’s name and namespace.
az identity federated-credential create \
--name managed-identity-test-fed-cred \
--identity-name managed-identity-test \
--resource-group mi-test-group \
--issuer https://oidc.prod-aks.azure.com/0eb05a4c-2d75-4568-ad04-0bb03f6a99b1/ \
--subject system:serviceaccount:platform:managed-identity-test
Remember the all-important trailing / in the OIDC URL.
I promised an image once the puzzle was complete, so here it is. Microsoft has much good documentation spread across numerous web pages, but it can still be tricky to understand what is happening where precisely.
The following diagram is what I pieced together from Microsoft’s more generally verbalized documentation here and here and here. The concept of federated credentials that are the basis of AAD Workload Identity supposedly works for more use cases than AKS pods. Examples that Microsoft uses include GitHub Actions or even Google Cloud.
I narrowed the scope to just the Azure Kubernetes use case I am discussing here, and, therefore, I omitted a few elements for simplicity.
The previous sections should already explain enough about what happens inside the Kubernetes cluster. The remaining question revolves around how authentication against AAD works. The microservice trapped in the Pod reads the federated token from the file mounted into the Pod. The application uses the federated token to authenticate against Azure AD and exchange it for an Azure AD token. At this point, the microservice can authenticate itself as the Managed Identity and access the Azure infrastructure to which the Managed Identity was granted access.
How exactly AAD draws the connection from the federated token to the Managed Identity is not exactly clear to me, but it is also not important. It would be obvious if it were a Service Principal, as they are part of AAD.
The federated token contains the following information, among which is the sub also present in the federated credential that connects the Kubernetes Service Account and Managed Identity.
{
"aud": [
"api://AzureADTokenExchange"
],
"exp": 1668430708,
"iat": 1668427108,
"iss": "https://oidc.prod-aks.azure.com/0eb05a4c-2d75-4568-ad04-0bb03f6a99b1/",
"kubernetes.io": {
"namespace": "platform",
"pod": {
"name": "managed-identity-test-5fd59c9658-cp6td",
"uid": "c0fd7bc6-6609-4016-b093-2a428e250c38"
},
"serviceaccount": {
"name": "managed-identity-test",
"uid": "598427ef-cb54-4f89-9604-a7f4d2645477"
}
},
"nbf": 1668427108,
"sub": "system:serviceaccount:platform:managed-identity-test"
}
For the curious, here is how the federated token differs from the default Service Account token automatically injected into every Pod.
{
"aud": [
"https://oidc.prod-aks.azure.com/0eb05a4c-2d75-4568-ad04-0bb03f6a99b1/",
"https://mi-test-aks-k8s-halo343.hcp.westeurope.azmk8s.io",
"\"mi-test-aks-k8s-halo343.hcp.westeurope.azmk8s.io\""
],
"exp": 1699963108,
"iat": 1668427108,
"iss": "https://oidc.prod-...
}
The value of aud is not the same, and everything else is (excluding the timestamps).
I certainly spend an awful lot of time explaining infrastructure as someone who prefers to code…
Speaking of which…
(Haha, segue!)
How would you take advantage of all the earlier configurations and setup? Let me explain the hard way to provide a complete picture, and then I will show you the more straightforward and practical solution. Either way, you need the azure-identity dependency. I based my demo application on Spring Boot and gave my Managed Identity read access to Key Vault secrets. Hence the azure-security-keyvault-secrets dependency.
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-sdk-bom</artifactId>
<version>1.2.7</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-security-keyvault-secrets</artifactId>
</dependency>
</dependencies>
To access Key Vault secrets, you need a SecretClient. You can get this from a SecretClientBuilder, which itself requires some form of authentication. The Azure SDK abstracts this as a TokenCredential.
Here is the bean definition.
private final IdentityConfig identityConfig;
@Bean
public SecretClient secretClient() {
// Make the vault configurable, of course.
return new SecretClientBuilder()
.vaultUrl("https://<vault-on-pandora>.vault.azure.net")
.credential(identityConfig.workloadIdentityCredential())
.buildClient();
}
As you now know, you require the federated token to authenticate your microservice against Azure AD. The hard way is to code this yourself and implement the TokenCredential interface.
@RequiredArgsConstructor
public class WorkloadIdentityCredential implements TokenCredential {
private final String clientID;
private final String tenantID;
private final String aadAuthority;
private final String federatedToken;
@Override
public Mono<AccessToken> getToken(TokenRequestContext requestContext) {
try {
var clientCredential = ClientCredentialFactory.createFromClientAssertion(
federatedToken);
var authority = String.format("%s%s", aadAuthority, tenantID);
var app = ConfidentialClientApplication.builder(clientID, clientCredential)
.authority(authority)
.build();
var scopes = new HashSet<>(requestContext.getScopes());
var clientCredentialParam = ClientCredentialParameters.builder(scopes)
.build();
var authResult = app.acquireToken(clientCredentialParam).get();
var expiresOnInstant = authResult.expiresOnDate().toInstant();
var expiresOn = OffsetDateTime.ofInstant(expiresOnInstant, ZoneOffset.UTC);
var accessToken = new AccessToken(authResult.accessToken(), expiresOn);
return Mono.just(accessToken);
} catch (Exception ex) {
return Mono.error(ex);
}
}
}
I based this code on an example Microsoft published for use with the Google Cloud. The federatedToken field contains the actual token, not the path to the file. The magic happens by calling app.acquireToken(clientCredentialParam).get();. To complete the picture, here is the bean definition.
@Slf4j
@Getter
@Configuration
public class IdentityConfig {
@Value("${AZURE_CLIENT_ID}")
private String clientId;
@Value("${AZURE_TENANT_ID}")
private String tenantId;
@Value("${AZURE_AUTHORITY_HOST}")
private String authorityHost;
@Value("${AZURE_FEDERATED_TOKEN_FILE}")
private String tokenFile;
@Bean
public TokenCredential workloadIdentityCredential() {
try {
var federatedToken = Files.readString(Paths.get(tokenFile));
return new WorkloadIdentityCredential(clientId, tenantId, authorityHost, federatedToken);
} catch (IOException e) {
log.error("Could not read Azure federated token from file '{}'. Cannot authenticate as Managed Identity " +
"'{}'.", tokenFile, clientId);
return null;
}
}
}
As you can see, I captured all environment variables injected by the webhook into fields that I use in the WorkloadIdentityCredential class.
The best way to do this in Azure is by utilizing the DefaultAzureCredential class. You can throw away the custom TokenCredential implementation like I would strawberries and simply define a bean like this.
@Bean
public TokenCredential defaultAzureCredential() {
return new DefaultAzureCredentialBuilder().build();
}
That’s it. And it is even more flexible. The bean definition for the Key Vault access becomes the following. You only swap the type of credential you are using but gain so much more in the process.
@Bean
public SecretClient secretClient() {
return new SecretClientBuilder()
.vaultUrl("https://<vault-on-pandora>.vault.azure.net")
.credential(identityConfig.defaultAzureCredential())
.buildClient();
}
What is it that you get for free? Less code to maintain, for one. More importantly, you cannot authenticate as a Managed Identity from your IDE. Therefore, local debugging is out of the question. The DefaultAzureCredential class solves this by checking for several configuration settings in a specific order. It looks at the environment variables, your az CLI login, and more. A plugin provides an Azure login context inside the IDE for IntelliJ IDEA users. Log in with your Azure account and debug your microservice without problems and without changing the code. You only need to ensure that your account has the same permissions as the Managed Identity would have.
I haven’t tested all Azure services, of course. Nobody can, as there simply are too many.
(Maybe Chuck Norris could 🤔.)
However, it should work with all services where the SDK supports a TokenCredential, and you can configure the Azure service with RBAC. One exception that I know of is Azure’s managed PostgreSQL offering. You can use Managed Credentials, but it is very different, and I cover that topic in a separate post. This one is already meaty enough.
I like this a lot. Looking at a recent project where we manually managed login credentials, connection strings, and more for several different environments, I cannot wait for Azure Active Directory Workload Identity (had to use the full name 😅) to become available for general production use. It can save so much time and effort and even prevent errors from copying the wrong credential. There aren’t any. This happens more often than you would think. The human element strikes at some point.
I hope this was helpful. Thank you for reading.
Delivering excellence in IoT. We are an IoT Solution Provider for Smart Products, Connected Vehicles, Smart City, Smart Energy and Smart Production.