producer consumer - How to achieve high speed processing using Azure Event Hub? -

-

i working on poc azure event hubs implement same our application.

quick brief on flow.

  • created tool read csv data local folder , send event hub.
  • we sending event data in batch event hub.
  • with 12 instance of tool (parallel), can send total of 600 000 lines of messages event hub within 1 min.
  • but, on receiver side, receive 600 000 lines of data, takes more 10 mins.

need achieve

  • i match/double egress speed on receiver process data. existing configuration

the configuration have made user are

  • tu - 10 1 event hub 32 partition.

  • coding logic goes same mentioned in msdn

  • only difference is, sending line of data in batch.
    eventprocessorhost options {maxbatchsize= 1000000,
    prefetchcount=1000000

share|improve question
up vote 1 down vote

to achieve higher egress rate (aka faster processing pipeline) in eventhubs:

  1. create scaled-out pipeline - each partition in eventhub unit-of-scale processing events out of eventhub. scale described (6lakh events per min --> 10k events per sec - 32 partitions - got right). make sure create many partitions envision pipeline need in near future. imagine analyzing traffic on highway , no. of lanes limitation amount of traffic.

  2. equal load distribution across partitions: if using sendtoaspecificpartition or sendusingpartitionkey - need take care of equal load distribution. if use eventhubclient.send(eventdatawithoutpartitionkey) - eventhubs service make sure of partitions equally loaded. if single eventhub partition heavily loaded - amount of time can process events on eventhub bound no. of events on partition.

  3. scale-out physical resources on receiver/eventprocessorhost: importantly network (sockets & bandwidth) & after-a-point, cpu & memory. use partitionmanageroptions.maxreceiveclients increase maximum number of eventhubclients (which has dedicated messagingfactory, maps 1 socket) created per eventprocessorhost instance. default 16.

let me know how went... :)

share|improve answer
    
hi sreeram, on point 1 : ingress taking 6 lacks event/minute 12 instance of sender. event hub still 32 partiotion. point 2 : using eventhubclient.send(eventdatawithoutpartitionkey) , event processor host can manage load. point 3: more details on messagefactory respect eventhub. please share sample example refer? [note]: using below constructor of eventprocessorhost public eventprocessorhost(string hostname, string eventhubpath, string consumergroupname, string eventhubconnectionstring, string storageconnectionstring); – à²…ನಿಲ್ aug 19 '16 @ 9:22
    
hi anil, had revisit point (3) - pl. take look. also, make sure eventhubs receive real bottleneck - remove blocking i/o calls in ieventprocessor.processeventsasync (eliminate possibility processeventsasync not contributing 10 min.. – sreeram aug 19 '16 @ 19:21
    
hi @sreeram , tried increasing tu 20 , found, 6 lacks messages processed in 5 min (earlier 10 min). increase event hub partition 100 (can create new 1 100 partition) , check load testing. there way increase/create new eh 100 partition? check higher possibility before going use application. – à²…ನಿಲ್ aug 23 '16 @ 5:46

your answer

 
discard

posting answer, agree privacy policy , terms of service.

not answer you're looking for? browse other questions tagged or ask own question.

Comments

Post a Comment