ExponentialBackoffPolicy should add randomness to backoff interval

[wheezil]

ExponentialBackoffPolicy.shouldRetry() uses a completely deterministic wait time. This can lead to continued failures in the face of multiple parallel processes launched more-or-less simultaneously, because each task will restart its attempt at the same time. While this case may seem contrived, it is exactly what happens when we attempt to launch 100s of tasks in Hadoop that perform parallel reads on a set of "part" files stored in ADL.

Suggested solution:

private void wait(int milliseconds) {
        if (milliseconds <= 0) {
            return;
        }
        try {
            Thread.sleep(milliseconds / 2 + randomInt(milliseconds / 2 + 1));

where randomInt() could be Random.nextInt() if a highly random seed were used.

[wheezil]

I'd like a second opinion about whether this will help, as I don't know how ADL counts its requests. The doc says "no more than 12000 reads/hour". But how long is the window used to accumulate the requests? If the window really is an hour long, then piling on 12000 requests all at once will render the ADL store unusable for nearly an hour. That's a really long time to wait.

[rahuldutta90]

@wheezil i will confirm the internal throttling logic and get back to you