O Sertão será Cloud

Facebook Youtube Instagram Linkedin

Unleashing the Power of Azure Event Hub with Azure Functions: Managed Identities with TypeScript!

Por Cláudio RaposoEm BlogPostou

Welcome to the seventh article in our series on integrating Azure Functions with Azure services, using the best programming practices! 🎉 We have already explored Azure Containers, Azure Key Vault, Azure Web PubSub, and Azure Service Bus. Now, let’s dive into Azure Event Hub, a large-scale data streaming service. 📊

Note: To see how to integrate with Azure Service Bus and Azure Web PubSub, see:

Now, speaking a little about Azure Event Hubs, it stands out as a real-time data ingestion and large-scale event streaming service. It is designed to facilitate the collection, processing, and analysis of large volumes of event data generated by applications, devices, and sensors. Here are some key points to understand Azure Event Hubs:

Main Features

  • High-Speed Data Ingestion: Event Hubs can ingest millions of events per second, allowing you to collect real-time data from various sources.
  • Data Buffering: It acts as a temporary buffer, storing events for a configurable period, allowing data consumers to process them at their own pace.
  • Partitioning: Events can be partitioned to enable parallel ingestion and horizontal scalability, which improves processing capacity.
  • Event Streaming: Besides collecting data, Event Hubs can stream events to multiple consumers, allowing different systems and services to consume data simultaneously.
  • Support for Various Platforms and Languages: You can send events to Event Hubs using various programming languages and platforms, including .NET, Java, Python, Node.js, and more.
  • Security and Compliance: Event Hubs offers robust security with authentication and authorization based on Microsoft Entra ID (formerly AAD) and SAS (Shared Access Signature) keys.

Use Cases

  • Application and Infrastructure Monitoring: Collect logs and metrics from servers, containers, and applications for real-time analysis and monitoring.
  • Real-Time Data Processing: Analyze data from IoT (Internet of Things) sensors, financial transactions, e-commerce, and more in real time.
  • ETL (Extract, Transform, Load): Integrate data from various sources, transform it as needed, and load it into storage systems or databases.
  • Clickstream Analysis: Track and analyze user clicks and interactions on websites and apps for behavior insights and personalization.
  • Integration with Other Azure Services: Event Hubs integrates well with other Azure services such as Azure Stream Analytics, Azure Functions, Azure Data Lake, and more to build complete data processing pipelines.

Azure Event Hubs Structure

  • Namespace: A top-level container for one or more Event Hubs.
  • Event Hub: A stream of event data within the namespace, similar to a topic in messaging systems.
  • Partitions: Subdivisions of an Event Hub that allow for parallel and independent ingestion of events.
  • Consumer Groups: Multiple consumer groups can read from the same Event Hub, each with its own reading position.

How It Works

  • Producers: Send data to the Event Hub. They can be applications, devices, or any system that produces events.
  • Event Hubs: Receives and stores events, distributing them across multiple partitions for scalability.
  • Consumers: Applications or services that read events from the Event Hub. They can process, analyze, or store the data as needed.
  • Processing and Storage: Consumed data can be processed in real time (e.g., using Azure Stream Analytics) or stored for later analysis (e.g., in Azure Data Lake or Azure SQL Database).

Azure Event Hubs is a powerful solution for handling large volumes of real-time data, offering scalability, security, and integration with other Azure services. It is ideal for scenarios that require rapid ingestion and parallel processing of data, allowing organizations to quickly turn large amounts of data into valuable insights.

Step 1: Environment Setup

Before you start, make sure you have the following dependencies installed:

npm install @azure/identity @azure/event-hubs

Note: If you need to see Steps 2 to 3, refer to the previous article on the topic:

How to Configure Azure Blob Service Client Applying SOLID Principles in an Azure Function with…

Step 4: Setting Up the Factory for Secret Clients

Now, let’s create a factory to instantiate the EventHubProducerClient using our CredentialProvider.

import { EventHubProducerClient } from "@azure/event-hubs";

/**
 * Fábrica para criar instâncias do EventHubProducerClient.
 */
class EventHubProducerClientFactory {
  private fullyQualifiedNamespace: string;
  private eventHubName: string;
  private credentialProvider: CredentialProvider;

  /**
   * Cria uma nova instância do EventHubProducerClientFactory.
   * @param {string} fullyQualifiedNamespace - O namespace totalmente qualificado do Azure Event Hub.
   * @param {string} eventHubName - O nome do Event Hub.
   * @param {CredentialProvider} credentialProvider - O provedor de credenciais a ser utilizado.
   */
  constructor(fullyQualifiedNamespace: string, eventHubName: string, credentialProvider: CredentialProvider) {
    this.fullyQualifiedNamespace = fullyQualifiedNamespace;
    this.eventHubName = eventHubName;
    this.credentialProvider = credentialProvider;
  }

  /**
   * Cria uma instância do EventHubProducerClient.
   * @returns {EventHubProducerClient} Uma instância do EventHubProducerClient configurada.
   */
  createEventHubProducerClient(): EventHubProducerClient {
    const credential = this.credentialProvider.getCredential();
    return new EventHubProducerClient(this.fullyQualifiedNamespace, this.eventHubName, credential);
  }
}

Step 5: Implementing in Azure Function

Now let’s see how we can integrate this structure into an Azure Function, demonstrating each authentication method.

Example 1: Azure Function with System-assigned Managed Identity

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { EventHubProducerClientFactory } from "./EventHubProducerClientFactory";
import { SystemAssignedManagedIdentityCredentialProvider } from "./SystemAssignedManagedIdentityCredentialProvider";

/**
 * Azure Function utilizando Managed Identity atribuída ao Sistema.
 * @param {Context} context - O contexto da função.
 * @param {HttpRequest} req - A requisição HTTP.
 * @returns {Promise<void>} Uma promessa que resolve quando a função é concluída.
 */
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
  const fullyQualifiedNamespace = process.env.AZURE_EVENTHUB_FULLYQUALIFIEDNAMESPACE;
  const eventHubName = "EVENT_HUB_NAME";
  const credentialProvider = new SystemAssignedManagedIdentityCredentialProvider();
  const producerClientFactory = new EventHubProducerClientFactory(fullyQualifiedNamespace, eventHubName, credentialProvider);
  
  const producerClient = producerClientFactory.createEventHubProducerClient();
  // Utilize o producerClient para enviar eventos para o Event Hub
  context.res = {
    body: "Azure Function utilizando Managed Identity atribuída ao Sistema para Acesso ao Event Hub"
  };
};

export default httpTrigger;

Example 2: Azure Function with User-assigned Managed Identity

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { EventHubProducerClientFactory } from "./EventHubProducerClientFactory";
import { UserAssignedManagedIdentityCredentialProvider } from "./UserAssignedManagedIdentityCredentialProvider";

/**
 * Azure Function utilizando Managed Identity atribuída ao Usuário.
 * @param {Context} context - O contexto da função.
 * @param {HttpRequest} req - A requisição HTTP.
 * @returns {Promise<void>} Uma promessa que resolve quando a função é concluída.
 */
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
  const fullyQualifiedNamespace = process.env.AZURE_EVENTHUB_FULLYQUALIFIEDNAMESPACE;
  const eventHubName = "EVENT HUB NAME";
  const clientId = process.env.AZURE_EVENTHUB_CLIENTID;
  const credentialProvider = new UserAssignedManagedIdentityCredentialProvider(clientId);
  const producerClientFactory = new EventHubProducerClientFactory(fullyQualifiedNamespace, eventHubName, credentialProvider);
  
  const producerClient = producerClientFactory.createEventHubProducerClient();
  // Utilize o producerClient para enviar eventos para o Event Hub
  context.res = {
    body: "Azure Function utilizando Managed Identity atribuída ao Usuário para Acesso ao Event Hub"
  };
};

export default httpTrigger;

Example 3: Azure Function with Service Principal

import { AzureFunction, Context, HttpRequest } from "@azure/functions";
import { EventHubProducerClientFactory } from "./EventHubProducerClientFactory";
import { ServicePrincipalCredentialProvider } from "./ServicePrincipalCredentialProvider";

/**
 * Azure Function utilizando Service Principal.
 * @param {Context} context - O contexto da função.
 * @param {HttpRequest} req - A requisição HTTP.
 * @returns {Promise<void>} Uma promessa que resolve quando a função é concluída.
 */
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
  const fullyQualifiedNamespace = process.env.AZURE_EVENTHUB_FULLYQUALIFIEDNAMESPACE;
  const eventHubName = "EVENT HUB NAME";
  const tenantId = process.env.AZURE_EVENTHUB_TENANTID;
  const clientId = process.env.AZURE_EVENTHUB_CLIENTID;
  const clientSecret = process.env.AZURE_EVENTHUB_CLIENTSECRET;
  const credentialProvider = new ServicePrincipalCredentialProvider(tenantId, clientId, clientSecret);
  const producerClientFactory = new EventHubProducerClientFactory(fullyQualifiedNamespace, eventHubName, credentialProvider);
  
  const producerClient = producerClientFactory.createEventHubProducerClient();
  // Utilize o producerClient para enviar eventos para o Event Hub
  context.res = {
    body: "Azure Function utilizando Service Principal para Acesso ao Event Hub"
  };
};

export default httpTrigger;

Conclusion

Congratulations! You have successfully configured Azure Event Hub using various authentication techniques in Azure Functions, applying best practices. Our modular approach facilitates code maintenance and extension. 🎉

References

Início Rápido – Criar uma conexão de serviço no aplicativo de funções por meio do portal do Azure

Unleashing the Power of Azure Event Hub with Azure Functions: Managed Identities with TypeScript! was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

Cláudio Raposo

Cláudio Raposo