It is important to consider that as sample rate increases, your computer has less time to process the same amount of data. For instance, at a sample rate of 48kHz, a buffer size of 128 samples will produce a latency of 2.66ms. At 96kHz, the same buffer size would produce a latency value of 1.55ms. At the higher rate, your computer may not be fast enough or have the resources available to play the audio without errors. Consequently, it is likely that you would need to increase the buffer size to allow your computer to work more efficiently. Note that if you are doubling the sample rate, for example, you can double the size of the buffer and maintain the same latency at the higher sample rate.