azure-eventhub

Looking for clarity on Event Hubs vs Topics in Azure Service Bus

孤者浪人 提交于 2019-11-29 20:37:31
I've been learning about Event Hubs and just want to get confirmation or correction on my perspective on Event Hubs? I’m used to leveraging retries, poison messages, at least once delivery and so on for normal enterprise messaging solutions, which Azure Service Bus Queues and Topics give me. It seems that Event Hubs is intended to provide a different tool for very high scale where you have to give up a little of the more “enterprise” features for much higher scale. Am I thinking about this correctly? Are there additional specifics I need to consider as well? I realize there could be some

Forcing EventProcessorHost to re-deliver failed Azure Event Hub eventData's to IEventProcessor.ProcessEvents method

风流意气都作罢 提交于 2019-11-29 15:55:22
问题 The application uses .NET 4.6.1 and the Microsoft.Azure.ServiceBus.EventProcessorHost nuget package v2.0.2, along with it's dependency WindowsAzure.ServiceBus package v3.0.1 to process Azure Event Hub messages. The application has an implementation of IEventProcessor . When an unhandled exception is thrown from the ProcessEventsAsync method the EventProcessorHost never re-sends those messages to the running instance of IEventProcessor . (Anecdotally, it will re-send if the hosting application

High throughput send to EventHubs resulting into MessagingException / TimeoutException / Server was unable to process the request errors

心已入冬 提交于 2019-11-29 04:51:34
We are experiencing lots of these exceptions sending events to EventHubs during peak traffic: "Failed to send event to EventHub. Exception : Microsoft.ServiceBus.Messaging.MessagingException: The server was unable to process the request; please retry the operation. If the problem persists, please contact your Service Bus administrator and provide the tracking id." or "Failed to send event to EventHub. Exception : System.TimeoutException: The operation did not complete within the allocated time " You can see it clearly here: As you can see, we got lots of Internal Errors, Server Busy Errors,

understanding check pointing in eventhub

风格不统一 提交于 2019-11-28 20:44:47
I want to ensure that, if my eventhub client crashes (currently a console application), it only picks up events it has not yet taken from the eventhub. One way to achieve this, is to exploit offsets. However, this (to my understanding) requires the client to store the latest offset (besides events do not necessarily seem to hit the foreach loop of the ProcessEventsAsync method ordered by SequenceNumber). An alternative, is to use checkpoints. I think they are persisted via the server (eventhub) using the provided storage account credentials. Is this correct? This is some preliminary code I am

Azure event hubs and multiple consumer groups

拈花ヽ惹草 提交于 2019-11-28 06:26:25
Need help on using Azure event hubs in the following scenario. I think consumer groups might be the right option for this scenario, but I was not able to find a concrete example online. Here is the rough description of the problem and the proposed solution using the event hubs ( I am not sure if this is the optimal solution. Will appreciate your feedback ) I have multiple event-sources that generate a lot of event data (telemetry data from sensors) which needs to be saved to our database and some analysis like running average, min-max should be performed in parallel . The sender can only send

How to set the EventProcessorHost to read events from now on (UTC)?

别来无恙 提交于 2019-11-28 01:30:22
问题 We are using the EventProcessorHost to receive events from Azure EventHubs. I've been unsuccessfully trying to configure it (through the EventProcessorOptions.InitialOffsetProvider) to read events from UTC now on but it always reads from the start of the feed. I am not saving checkpoints (and I even deleted the BLOB container created). This is how I am setting it: DateTime startDate = DateTime.UtcNow; var epo = new EventProcessorOptions { MaxBatchSize = 100, PrefetchCount = 100,

What Azure .NET SDK EventHubClient instance methods are threadsafe?

纵然是瞬间 提交于 2019-11-28 00:23:04
I'm writing code that will be publishing messages from multiple threads to an Azure Event Hub in C# using the EventHubClient . The documentation for EventHubClient contains the fairly standard boiler plate. "Any public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe." There is no additional documentation as to thread safety in any of the four send methods I would most expect to be thread safe. Were I to believe that the send methods are not threadsafe then I would end up creating a new EventHubClient instance each

What is causing Azure Event Hubs ReceiverDisconnectedException/LeaseLostException?

走远了吗. 提交于 2019-11-27 19:06:11
I'm receiving events from an EventHub using EventProcessorHost and an IEventProcessor class (call it: MyEventProcessor). I scale this out to two servers by running my EPH on both servers, and having them connect to the Hub using the same ConsumerGroup, but unique hostName's (using the machine name). The problem is: at random hours of the day/night, the app logs this: Exception information: Exception type: ReceiverDisconnectedException Exception message: New receiver with higher epoch of '186' is created hence current receiver with epoch '186' is getting disconnected. If you are recreating the

understanding check pointing in eventhub

本秂侑毒 提交于 2019-11-27 13:07:43
问题 I want to ensure that, if my eventhub client crashes (currently a console application), it only picks up events it has not yet taken from the eventhub. One way to achieve this, is to exploit offsets. However, this (to my understanding) requires the client to store the latest offset (besides events do not necessarily seem to hit the foreach loop of the ProcessEventsAsync method ordered by SequenceNumber). An alternative, is to use checkpoints. I think they are persisted via the server

Azure Functions Event Hub trigger bindings

99封情书 提交于 2019-11-27 08:13:41
问题 Just have a couple of questions regarding the usage of Azure Functions with an EventHub in an IoT scenario. EventHub has partitions. Typically messages from a specific device go to the same partition. How are the instances of an Azure Function distributed across EventHub partitions? Is it based on the performance? In case one instance of an Azure Function manages to process events from all partitions then it is enough otherwise one might end up with one instance of an Azure Function per