!h*BSrSSKJr SSKrSSKrSSKrSSKJr SSKJrJ r J r J r SSK J r SSKJr SSKJrJrJrJrJrJrJr SS KJr SS KJrJrJr SS KJr SS K J!r! SS K"J#r#J$r$ SSK%J&r&J'r' \(aSSKJ(r(J)r) SSK*J+r+ SSK,J-r-J.r. \."S5r/\"S5r0\"S5r1Sr2S0Sjr3S1Sjr4S2Sjr5S3Sjr6\S4Sj5r7\S5Sj5r7\S6Sj5r7S7Sjr7S8Sjr8"SS \ \\05r9S9S!jr:S:S"jr;S;S#jr<SS>S&jr?\"S'\\\\4S(9r@\S?S)j5rA\S@S*j5rASAS+jrASBS,jrBSCS-jrCSDS.jrDSES/jrEg)Fz5 Helper functions for dealing with Twisted deferreds ) annotationsN)Future) Awaitable CoroutineIterableIteratorwraps) CoroutineType) TYPE_CHECKINGAnyGenericTypeVarUnioncastoverload)defer)Deferred DeferredListensureDeferred) Cooperatorfailure) IgnoreRequestScrapyDeprecationWarning)_get_asyncio_event_loopis_asyncio_reactor_installed) AsyncIteratorCallable)Failure) Concatenate ParamSpec_P_T_T2g?chSSKJn [5nUR[UR U5 U$)zSame as twisted.internet.defer.fail but delay calling errback until next reactor loop It delays by 100ms so reactor has a chance to go through readers and writers before attending pending delayed calls, so do not set delay to zero. rreactor)twisted.internetr(r callLater _DEFER_DELAYerrback)_failurer(ds Y/root/1688_scrapy/alibaba-scraper/venv/lib/python3.13/site-packages/scrapy/utils/defer.py defer_failr0.s))zA lAIIx8 HchSSKJn [5nUR[UR U5 U$)zSame as twisted.internet.defer.succeed but delay calling callback until next reactor loop It delays by 100ms so reactor has a chance to go through readers and writers before attending pending delayed calls, so do not set delay to zero. rr'r)r(rr*r+callback)resultr(r.s r/ defer_succeedr6<s))jA lAJJ7 Hr1chSSKJn [5nUR[UR S5 U$)zNLike ``defer_succeed`` and ``defer_fail`` but doesn't call any real callbacks.rr'Nr3)r(r.s r/ _defer_sleepr8Js'( A lAJJ5 Hr1c[U[5(aU$[U[R5(a [ U5$[ U5$N) isinstancerrr r0r6)r5s r/ defer_resultr<Ss;&(## &'//**&!!   r1cgr:fargskws r/mustbe_deferredrC[r1cgr:r>r?s r/rCrCas r1cgr:r>r?s r/rCrCirDr1cU"U0UD6n[ U5$![a)n[[R"U55sSnA$SnAf[a! [[R"55s$f=f)zfSame as twisted.internet.defer.maybeDeferred, but delay calling callback/errback to next reactor loop N)rr0rr Exceptionr<)r@rArBr5es r/rCrCosa-DB   .'//!,-- -'//+,,-s A1AA1+A10A1c^^^[5nUUU4SjU5n[[U5Vs/sHouRU5PM sn5$s snf)zExecute a callable over the objects in the given iterable, in parallel, using no more than ``count`` concurrent calls. Taken from: https://jcalderone.livejournal.com/24285.html c3<># UHnT"U/TQ70TD6v M g7fr:r>).0elemrAcallablenameds r/ parallel..s Ohd8D94959hs)rrrange coiterate)iterablecountrNrArOcoopwork_s ``` r/parallelrYs> # 6837r1cPSUlUR"U/URQ70URD6nURR S5n[ U[5(aURU5 OURS5 UR(aUR5 gg)Nr) rdrNr`rarcpopr;r chainDeferredr4 _call_anext)rer5callable_resultr.s r/ _callback!_AsyncCooperatorAdapter._callbacks#--  '' +/+?+?   " " & &q ) ox 0 0  ) )! , JJt   ! !     "r1cSUlUR"[5 SUlURHnUR S5 M g)NT)rdtrapStopAsyncIterationrbrcr4)rerr.s r/_errback _AsyncCooperatorAdapter._errbacks<# '( ''A JJt (r1c[URR55UlURR UR UR 5 gr:)deferred_from_coror_ __anext__rd addCallbacksrnrs)res r/rl#_AsyncCooperatorAdapter._call_anexts<11I1I1KL ((Gr1cUR(a[e[5nURR U5 UR (dUR 5 U$r:)rb StopIterationrrcappendrdrl)rer.s r/__next__ _AsyncCooperatorAdapter.__next__sF == #: %%a(""    r1)r_rdrNr`rarbrcN)rfAsyncIterator[_T]rN3Callable[Concatenate[_T, _P], Deferred[Any] | None]r`_P.argsra _P.kwargs)r5r$returnNone)rr rr)rr)r Deferred[Any]) __name__ __module__ __qualname____firstlineno____doc__rgrnrsrlr}__static_attributes__r>r1r/r[r[sI+Z 8$ 8F 8 8 % 8 H r1r[c[5n[X/UQ70UD6n[[U5Vs/sHouR U5PM sn5nU$s snf)z)Like ``parallel`` but for async iterators)rr[rrRrS) async_iterablerUrNrArOrVrWrXdls r/parallel_asyncrs` )rrs r/"process_parallel..BsQ6GQtQ6G6Gsc.URR$r:)value subFailurers r/rprocess_parallel..ebDs}}'''r1)rr rr )rsucceedrrr) rrrrBrdfdsr.d2rs r/process_parallelr5sxDM M9aEMM% , ,Q 9 9b 99D M*6 t4+A mm,GHB(MM" I Ns4A0c/# [U5n[U5v M![a g[a" U"[R "5/UQ70UD6 N8f=f7f)zOWraps an iterable calling an errback if an error is caught while iterating it. N)iternextr{rHrr )rTr,rrBits r/ iter_errbackrKsZ hB  1r(N    1 GOO% 0 0R 0 1s/ A A AA(AAAAc# UR5nUR5IShvN 7v MN ![a g[a" U"[R "5/UQ70UD6 N:f=f7f)zWraps an async iterable calling an errback if an error is caught while iterating it. Similar to :func:`scrapy.utils.defer.iter_errback`. N)r^rwrrrHrr )rfr,rrBrs r/ aiter_errbackr^sf    B  1 & & &!   1 GOO% 0 0R 0 1s<A.424A.4 A+A.(A+(A.*A++A._CT)boundcgr:r>os r/rvrvts,/r1cgr:r>rs r/rvrvxs%(r1cx[U[5(aU$[R"U5(d[R "U5(al[ 5(d+[[[[[[4U55$[5n[R"[R"XS95$U$)ztConverts a coroutine or other awaitable object into a Deferred, or returns the object as is if it isn't a coroutine.)loop)r;rasyncioisfutureinspect isawaitablerrrrr r fromFuture ensure_future)r event_loops r/rvrv|s!Xg11!44+--"$y31C'Da"HI I,. ""7#8#8#LMM Hr1c4^[T5SU4Sjj5nU$)a!Converts a coroutine function into a function that returns a Deferred. The coroutine function will be called at the time when the wrapper is called. Wrapper args will be passed to it. This is useful for callback chains, as callback functions are called with the previous callback result. c&>[T"U0UD65$r:)rv) coro_args coro_kwargscoro_fs r/r@!deferred_f_from_coro_f..fs!&)"C{"CDDr1)rrrrrr r )rr@s` r/deferred_f_from_coro_frs$ 6]EE Hr1cU"U0UD6n[ U[5(aU$[R"U5(d[R"U5(a [U5$[ U[R5(a[R"U5$[R"U5$! [R"[R"[R S95s$=f)zGCopy of defer.maybeDeferred that also converts coroutines to Deferreds.) captureVars) rfailrr rdebugr;rrrrrvr)r@rArBr5s r/maybeDeferred_cororsGDB&(## 7#6#6v#>#>!&))&'//**zz&!! ==  Gzz'//hnnEFFs B--8C'c4UR[55$)a .. versionadded:: 2.6.0 Return an :class:`asyncio.Future` object that wraps *d*. When :ref:`using the asyncio reactor `, you cannot await on :class:`~twisted.internet.defer.Deferred` objects from :ref:`Scrapy callables defined as coroutines `, you can only await on ``Future`` objects. Wrapping ``Deferred`` objects into ``Future`` objects allows you to wait on them:: class MySpider(Spider): ... async def parse(self, response): additional_request = scrapy.Request('https://example.org/price') deferred = self.crawler.engine.download(additional_request) additional_response = await deferred_to_future(deferred) )asFuturerr.s r/deferred_to_futurers& ::-/ 00r1c:[5(dU$[U5$)a .. versionadded:: 2.6.0 Return *d* as an object that can be awaited from a :ref:`Scrapy callable defined as a coroutine `. What you can await in Scrapy callables defined as coroutines depends on the value of :setting:`TWISTED_REACTOR`: - When :ref:`using the asyncio reactor `, you can only await on :class:`asyncio.Future` objects. - When not using the asyncio reactor, you can only await on :class:`~twisted.internet.defer.Deferred` objects. If you want to write code that uses ``Deferred`` objects but works with any reactor, use this function on all ``Deferred`` objects:: class MySpider(Spider): ... async def parse(self, response): additional_request = scrapy.Request('https://example.org/price') deferred = self.crawler.engine.download(additional_request) additional_response = await maybe_deferred_to_future(deferred) )rrrs r/maybe_deferred_to_futurers4 ( ) ) a  r1)r-r rr)r5r$r Deferred[_T])rzDeferred[None])r5r rr)r@Callable[_P, Deferred[_T]]rArrBrrr)r@z/Callable[_P, Coroutine[Deferred[Any], Any, _T]]rArrBrrr)r@zCallable[_P, _T]rArrBrrr)r@zCCallable[_P, Deferred[_T] | Coroutine[Deferred[Any], Any, _T] | _T]rArrBrrr) rT Iterable[_T]rUintrNz"Callable[Concatenate[_T, _P], _T2]rArrOrrz*Deferred[list[tuple[bool, Iterator[_T2]]]]) rrrUrrNrrArrOrrz4Deferred[list[tuple[bool, Iterator[Deferred[Any]]]]]) rz+Iterable[Callable[Concatenate[_T, _P], _T]]rr$rrrBrrr) rz,Iterable[Callable[Concatenate[_T, _P], Any]]rz1Iterable[Callable[Concatenate[Failure, _P], Any]]rr rrrBrrr) rz,Iterable[Callable[Concatenate[_T, _P], _T2]]rr$rrrBrrzDeferred[list[_T2]]) rTrr,'Callable[Concatenate[Failure, _P], Any]rrrBrrr) rfrr,rrrrBrrr)rrrr)rr$rr$)rr$rz Deferred | _T)rz%Callable[_P, Coroutine[Any, Any, _T]]rr)r@zCallable[_P, Any]rArrBrrr)r.rrz Future[_T])r.rrzDeferred[_T] | Future[_T])Fr __future__rrrrrcollections.abcrrrr functoolsr typesr typingr r rrrrrr)rtwisted.internet.deferrrrtwisted.internet.taskrtwisted.pythonrscrapy.exceptionsrrscrapy.utils.reactorrrrrtwisted.python.failurer typing_extensionsr!r"r#r$r%r+r0r6r8r<rCrYr[rrrrrrrrvrrrrr>r1r/rs}#DDNNN" -"EV7.9 4B T] en      ! !*19B   6      '/8   J      (FF F1F  F  F 0 F"eh eP% B     : $  :             ; ?          2;      ,11 41 1  1  1&1 1 41 1  1  1&e5M6!ABC / / ( (   1    !!!(!09!!$1,!r1