Azure Key Vault Event - Is it possible to subscribe to Event Grid Topic (not system event grid topic)?

Question

I can create an event subscription at Azure Key Vault but it allows only system event grid topic as opposed to custom event grid topic. My preference is custom event grid topic because I can assign a managed identity and grant necessary RBAC to the managed identity.

Is it possible to configure Azure Key Vault to send events to custom event grid topic?

Here is a sample custom event grid topic:

{
    "name": "demoeventsubscription",
    "properties": {
        "topic": "/subscriptions/my-subscription-id/resourceGroups/EventGrids/providers/Microsoft.EventGrid/topics/kvTpoic",
        "destination": {
            "endpointType": "AzureFunction",
            "properties": {
                "resourceId": "/subscriptions/my-subscription-id/resourceGroups/aspnet4you/providers/Microsoft.Web/sites/afa-aspnet4you/functions/EventsProcessor",
                "maxEventsPerBatch": 1,
                "preferredBatchSizeInKilobytes": 64
            }
        },
        "filter": {
            "includedEventTypes": [
                "Microsoft.KeyVault.SecretNewVersionCreated"
            ],
            "advancedFilters": []
        },
        "labels": [],
        "eventDeliverySchema": "EventGridSchema"
    }
}

Answer 1

@MayankBargali-MSFT - Thanks for checking-in the GitHub reference here. You are correct, Azure resources (ie key vault, storage, etc.) are not currently designed to use custom event grid topic. Those resources are pre-configured for system event grid topic which does not allow customer to attach identity and without identity customer can't protect target or destination resource (subscriber of events) from unauthorized access.

As customer my expectation is, Azure would either allow custom event grid topic or modify the capability of system event grid topic to attach managed identity. We would be able to use RBAC on destination resource (azure function in my case) to control security on the managed identity.

For everyone's information, I used the following azure function to verify system event grid topic does not pass any identity in request header when invoking azure function (webhook)-

using System;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventGrid;
using Microsoft.Azure.EventGrid.Models;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Queue;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;

namespace AzureFunctionAppsCore
{
    public static class SampleEventGridConsumer
    {
        [FunctionName("SampleEventGridConsumer")]
        public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequestMessage req, ILogger log)
        {
            log.LogInformation($"C# HTTP trigger function begun");
            string response = string.Empty;
            

            try
            {
                JArray requestHeaders = Utils.GetIpFromRequestHeadersV2(req);
                log.LogInformation(requestHeaders.ToString());

                string requestContent = req.Content.ReadAsStringAsync().GetAwaiter().GetResult();
                log.LogInformation($"Received events: {requestContent}");

                // Get the event type dynamically so that we can add/update custom event mappings to  EventGridSubscriber
                // Keep in mind, this is designed for Key Vault event type schema only.
                KeyVaultEvent[] dynamicEventsObject = JsonConvert.DeserializeObject<KeyVaultEvent[]>(requestContent);
                
                EventGridSubscriber eventGridSubscriber = new EventGridSubscriber();

                foreach(KeyVaultEvent kve in dynamicEventsObject)
                {
                    eventGridSubscriber.AddOrUpdateCustomEventMapping(kve.eventType, typeof(KeyVaultEventData));
                }
                
                EventGridEvent[] eventGridEvents = eventGridSubscriber.DeserializeEventGridEvents(requestContent);

                foreach (EventGridEvent eventGridEvent in eventGridEvents)
                {
                    if (eventGridEvent.Data is SubscriptionValidationEventData)
                    {
                        var eventData = (SubscriptionValidationEventData)eventGridEvent.Data;
                        log.LogInformation($"Got SubscriptionValidation event data, validationCode: {eventData.ValidationCode},  validationUrl: {eventData.ValidationUrl}, topic: {eventGridEvent.Topic}");
                        // Do any additional validation (as required) such as validating that the Azure resource ID of the topic matches
                        // the expected topic and then return back the below response
                        var responseData = new SubscriptionValidationResponse()
                        {
                            ValidationResponse = eventData.ValidationCode
                        };



                        log.LogInformation($"Sending ValidationResponse: {responseData.ValidationResponse} and HttpStatusCode : {HttpStatusCode.OK}.");
                        
                        return new HttpResponseMessage(HttpStatusCode.OK)
                        {
                            Content = new StringContent(JsonConvert.SerializeObject(responseData), Encoding.UTF8, "application/json")
                        };
                    }
                    else if (eventGridEvent.Data is StorageBlobCreatedEventData)
                    {
                        var eventData = (StorageBlobCreatedEventData)eventGridEvent.Data;
                        log.LogInformation($"Got BlobCreated event data, blob URI {eventData.Url}");
                    }
                    else if (eventGridEvent.Data is KeyVaultEventData)
                    {
                        var eventData = (KeyVaultEventData)eventGridEvent.Data;
                        log.LogInformation($"Got KeyVaultEvent event data, Subject is {eventGridEvent.Subject}");

                        var connectionString = Config.GetEnvironmentVariable("AzureWebJobsStorage");

                        // Retrieve storage account from connection string.
                        CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);

                        // Create the queue client.
                        CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

                        // Retrieve a reference to a container.
                        CloudQueue queue = queueClient.GetQueueReference("eventgridqueue");

                        // Create the queue if it doesn't already exist
                        await queue.CreateIfNotExistsAsync();

                        // Create a message and add it to the queue.
                        CloudQueueMessage message = new CloudQueueMessage(JsonConvert.SerializeObject(eventGridEvent));
                        await queue.AddMessageAsync(message);

                    }
                }
            }
            catch(Exception ex)
            {
                log.LogError(ex.ToString());
            }


            return new HttpResponseMessage(HttpStatusCode.OK)
            {
                Content = new StringContent("Ok", Encoding.UTF8, "application/json")
            };
        }
    }

    public class KeyVaultEvent
    {
        public string id { get; set; }
        public string topic { get; set; }
        public string subject { get; set; }
        public string eventType { get; set; }
        public DateTime eventTime { get; set; }
        public KeyVaultEventData data { get; set; }
        public string dataVersion { get; set; }
        public string metadataVersion { get; set; }
    }

    public class KeyVaultEventData
    {
        public string Id { get; set; }
        public string vaultName { get; set; }
        public string objectType { get; set; }
        public string objectName { get; set; }
        public string version { get; set; }
        public string nbf { get; set; }
        public string exp { get; set; }
    }
}

Helper function:

public static JArray GetIpFromRequestHeadersV2(HttpRequestMessage request)
        {
            JArray values = new JArray();

            foreach(KeyValuePair<string, IEnumerable<string>> kvp in request.Headers)
            {
                values.Add(JObject.FromObject(new NameValuePair(kvp.Key, kvp.Value.FirstOrDefault())));
            }

            return values;
        }

Sharing project dependency of my .net core 3.1, in case you wanted to run locally:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <AzureFunctionsVersion>v3</AzureFunctionsVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BouncyCastle.NetCore" Version="1.8.6" />
    <PackageReference Include="Microsoft.Azure.EventGrid" Version="3.2.0" />
    <PackageReference Include="Microsoft.Azure.Management.Fluent" Version="1.34.0" />
    <PackageReference Include="Microsoft.Azure.Management.ResourceManager.Fluent" Version="1.34.0" />
    <PackageReference Include="Microsoft.Azure.OperationalInsights" Version="0.10.0-preview" />
    <PackageReference Include="Microsoft.Azure.Services.AppAuthentication" Version="1.5.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventGrid" Version="2.1.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.SendGrid" Version="3.0.0" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.Storage" Version="3.0.0" />
    <PackageReference Include="Microsoft.IdentityModel.Clients.ActiveDirectory" Version="5.2.0" />
    <PackageReference Include="Microsoft.IdentityModel.Tokens.Saml" Version="5.6.0" />
    <PackageReference Include="Microsoft.NET.Sdk.Functions" Version="3.0.3" />
    <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="5.6.0" />
  </ItemGroup>
  <ItemGroup>
    <None Update="host.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>
    <None Update="local.settings.json">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
      <CopyToPublishDirectory>Never</CopyToPublishDirectory>
    </None>
  </ItemGroup>
</Project>

So, what are the headers passed by system event grid topic when it invokes azure function?

[{
        "name": "Accept-Encoding",
        "value": "gzip"
    }, {
        "name": "Connection",
        "value": "Keep-Alive"
    }, {
        "name": "Host",
        "value": "afa-aspnet4you.azurewebsites.net"
    }, {
        "name": "Max-Forwards",
        "value": "10"
    }, {
        "name": "aeg-subscription-name",
        "value": "MYSUBSCRIPTION2"
    }, {
        "name": "aeg-delivery-count",
        "value": "0"
    }, {
        "name": "aeg-data-version",
        "value": "1"
    }, {
        "name": "aeg-metadata-version",
        "value": "1"
    }, {
        "name": "aeg-event-type",
        "value": "Notification"
    }, {
        "name": "X-WAWS-Unencoded-URL",
        "value": "/api/SampleEventGridConsumer?code=removed=="
    }, {
        "name": "CLIENT-IP",
        "value": "52.154.68.16:58880"
    }, {
        "name": "X-ARR-LOG-ID",
        "value": "1e5b1918-9486-4b41-ab1d-7a644bc263d2"
    }, {
        "name": "DISGUISED-HOST",
        "value": "afa-aspnet4you.azurewebsites.net"
    }, {
        "name": "X-SITE-DEPLOYMENT-ID",
        "value": "afa-aspnet4you"
    }, {
        "name": "WAS-DEFAULT-HOSTNAME",
        "value": "afa-aspnet4you.azurewebsites.net"
    }, {
        "name": "X-Original-URL",
        "value": "/api/SampleEventGridConsumer?code=removed=="
    }, {
        "name": "X-Forwarded-For",
        "value": "52.154.68.16:58880"
    }, {
        "name": "X-ARR-SSL",
        "value": "2048|256|C=US, O=Microsoft Corporation, CN=Microsoft RSA TLS CA 01|CN=*.azurewebsites.net"
    }, {
        "name": "X-Forwarded-Proto",
        "value": "https"
    }, {
        "name": "X-AppService-Proto",
        "value": "https"
    }, {
        "name": "X-Forwarded-TlsVersion",
        "value": "1.2"
    }
]

Azure function uses api key but it is shared and not considered true security in production system. System topic is not sending any header that we can use to find any identity.

Couple of events processed by azure function shown here-

[{
        "id": "2ff9617d-e498-4527-8467-d36eeff6b94b",
        "topic": "/subscriptions/truncated/resourceGroups/key-vaults/providers/Microsoft.KeyVault/vaults/aspnet4you-keyvault",
        "subject": "TexasSnow",
        "eventType": "Microsoft.KeyVault.SecretNewVersionCreated",
        "data": {
            "Id": "https://aspnet4you-keyvault.vault.azure.net/secrets/TexasSnow/106b1d8450e6404bb323abf650c81496",
            "VaultName": "aspnet4you-keyvault",
            "ObjectType": "Secret",
            "ObjectName": "TexasSnow",
            "Version": "106b1d8450e6404bb323abf650c81496",
            "NBF": null,
            "EXP": null
        },
        "dataVersion": "1",
        "metadataVersion": "1",
        "eventTime": "2021-02-20T06:50:08.6207125Z"
    }
]

[
    {
        "id": "996c900f-b697-4754-916c-67e485e141f9",
        "topic": "/subscriptions/b63613a2-9fc8-47ad-a65c-e1d1eba108be/resourceGroups/key-vaults/providers/Microsoft.KeyVault/vaults/aspnet4you-keyvault",
        "subject": "EventTestKey",
        "eventType": "Microsoft.KeyVault.KeyNewVersionCreated",
        "data": {
            "Id": "https://aspnet4you-keyvault.vault.azure.net/keys/EventTestKey/22f3358955174cff9f5d5f24f5f2ecb0",
            "VaultName": "aspnet4you-keyvault",
            "ObjectType": "Key",
            "ObjectName": "EventTestKey",
            "Version": "22f3358955174cff9f5d5f24f5f2ecb0",
            "NBF": null,
            "EXP": null
        },
        "dataVersion": "1",
        "metadataVersion": "1",
        "eventTime": "2021-02-20T20:08:00.4520828Z"
    }
]

So, what's next? As @MayankBargali-MSFT said, azure is working as designed. I have sent a feature request to feedback.azure.com .