golang + redis concurrency scheduler performance issue

Question

I write a simple concurrency scheduler, but it seems to have a performance issue at a high level concurrency.

Here is the code (scheduler + concurrent rate limiter test)

Answer 1

At a surface level the only thing that I have questions about is ordering of incrementing wait group and enqueing the work:

func (s *Scheduler) Enqueue(req interface{}) {
    select {
    case s.reqChan <- req:
        s.wg.Add(1)
    }
}

I don't think the above will cause much problem in practice with this large of workload, but I think it may be a logical race condition. At lower levels of concurrency and smaller work sizes, it may enqueue a message, conext switch to a goroutine that starts work on that message, THEN the work in the wait group.

---

Next are you sure process method is threadsafe?? I'd assume so based on the redis go documentation , does running with go run -race have any output?

---

At some point It's completely reasonable and expected for performance to drop off. I would recommend starting performance tests to see where latency and throughput start to drop off:

maybe a pool of 10, 100, 500, 1000, 2500, 5000, 10000, or whatever makes sense. IMO it looks like there are 3 important variables to tune:

• Worker pool size
• Work Queue Buffer Size
• Redis MaxActive

The biggest thing that jumps out is that it looks like redis.Pool is configured to allow an unbounded number of connections:

 pool := &redis.Pool{
        MaxIdle:     50,
        IdleTimeout: 240 * time.Second,
        TestOnBorrow: func(c redis.Conn, t time.Time) error {
            _, err := c.Do("PING")
            return err
        },
        Dial: func() (redis.Conn, error) {
            return dial("tcp", address, password)
        },
    }

// Maximum number of connections allocated by the pool at a given time. // When zero, there is no limit on the number of connections in the pool. MaxActive int

---

I would personally try to understand where and when performance starts to drop off with respect to the size of your worker pool. This might make it easier to understand what your program is constrained by.