dZddlZddlZddlZddlZddlZddlmZddlm Z Gdde Z dZ Gdd e Z d Zd d eje e d fdZy)aHRetry decorators for calls raising exceptions. This module is used mostly to decorate all integration points where the code makes calls to remote services. Searching through the code base for @retry should find all such places. For this reason even places where retry is not needed right now use a @retry.no_retries decorator. N) _exceptions)reraisec$eZdZdZ ddZdZy)FuzzedExponentialIntervalsa<Iterable for intervals that are exponentially spaced, with fuzzing. On iteration, yields retry interval lengths, in seconds. Every iteration over this iterable will yield differently fuzzed interval lengths, as long as fuzz is nonzero. Args: initial_delay_secs: The delay before the first retry, in seconds. num_retries: The total number of times to retry. factor: The exponential factor to use on subsequent retries. Default is 2 (doubling). fuzz: A value between 0 and 1, indicating the fraction of fuzz. For a given delay d, the fuzzed delay is randomly chosen between [(1 - fuzz) * d, d]. max_delay_sec: Maximum delay (in seconds). After this limit is reached, further tries use max_delay_sec instead of exponentially increasing the time. Defaults to 5 minutes. c||_||_||_d|cxkrdkstdtd||_||_y)Nrz.Fuzz parameter expected to be in [0, 1] range.)_initial_delay_secs _num_retries_factor ValueError_fuzz_max_delay_secs)selfinitial_delay_secs num_retriesfactorfuzzmax_delay_secss .lib/third_party/ml_sdk/cloud/ml/util/_retry.py__init__z#FuzzedExponentialIntervals.__init__5sR 2D#DDL >> G HH  G HHDJ)Dc#6Kt|j|j}t|jD]]}d|j z t j |j zz}||zt|j||jz}_yw)Nr)minrr ranger r randomr )rcurrent_delay_secs_fuzz_multipliers r__iter__z#FuzzedExponentialIntervals.__iter__CsT1143K3KL 4$$ %DJJ4::)EEo  00t331DLL@B&sBBN)?)__name__ __module__ __qualname____doc__rrrrrr!s, *BrrcZt|tjr|jdk\ryyy)zzno_retries..bsrN) retry_filterclock)with_exponential_backoff)funs r no_retriesr;`s K !d KC PPr rTc&fd}|S)aDecorator with arguments that control the retry logic. Args: num_retries: The total number of times to retry. initial_delay_secs: The delay before the first retry, in seconds. logger: A callable used to report en exception. Must have the same signature as functions in the standard logging module. The default is logging.warning. retry_filter: A callable getting the exception raised and returning True if the retry should happen. For instance we do not want to retry on 404 Http errors most of the time. The default value will return true for server errors (HTTP status code >= 500) and non Http errors. clock: A clock object implementing a sleep method. The default clock will use time.sleep(). fuzz: True if the delay should be fuzzed (default). During testing False can be used so that the delays are not randomized. Returns: As per Python decorators with arguments pattern returns a decorator for the function which in turn will return the wrapped (decorated) function. The decorator is intended to be used on callables that make HTTP or RPC requests that can temporarily timeout or have transient errors. For instance the make_http_request() call below will be retried 16 times with exponential backoff and fuzzing of the delay interval (default settings). from cloudml.util import retry # ... @retry.with_exponential_backoff() make_http_request(args) cTttrdndfd}|S)zCThe real decorator whose purpose is to return the wrapped function.r!r)rc  |i|S#t$r5}|stjd} t }n%#t$rt t |||YnwxYwdtdtdjtj|j|djtj|j|tjdkrtj d}n.#tjdkrtj d}wxYwYd}~nd}~wwxYwM)Nr zRetry with exponential backoff: waiting for %s seconds before retrying %s because we caught exception: %s Traceback for above exception (most recent call last): %sr#)r) Exceptionsysexc_infonext StopIterationrtypegetattrstrjoin tracebackformat_exception_only __class__ format_tbr2 version_info exc_clear) argskwargsexn exn_tracebacksleep_intervalr8r:loggerr7retry_intervalss rwrapperzAwith_exponential_backoff..real_decorator..wrappers'  !d%f% % !c" ,,.+- ! 5#O4n  5d3im4 5 MZS2 77 sKL ++M:; = KK ' &(mmo M&(mmo MM; ! sJ E  E ADA&#D%A&&BD)*E+D>>EE )iterr) r:rXrWr8rrrVrr7s ` @rreal_decoratorz0with_exponential_backoff..real_decorators6"  #1 FGO!!!!F Nrr')rrrVr7r8rrZs`````` rr9r9esL**X r)r&loggingrrCr1rKgoogle.cloud.ml.utilrsixrobjectrr-r/r;warningr9r'rrr`sg ,(B(BV FQ *,01$+OO*G#(7"& Rr