!hcMSSKJr SSKrSSKrSSKrSSKrSSKJrJrJ r J r SSK J r J r JrJr SSKJr SSKJrJr SSKJr SSKJr SS KJr SS KJr SS KJrJrJ r SS K!J"r" SS K#J$r$J%r%J&r&J'r'J(r(J)r) SSK*J+r+J,r, SSK-J.r.J/r/ SSK0J1r1J2r2J3r3J4r4 \(a SSK5J6r6J7r7 SSK8J9r9 SSK:J;r; SSKJ?r? \R"\A5rB\ "S5rC"SS5rD"SS5rE"SS\E5rFg)) annotationsN) TYPE_CHECKINGAnyTypeVarcast)Deferred DeferredListinlineCallbacks maybeDeferred) verifyClass)Spidersignals) AddonManagerExecutionEngine)ExtensionManager) ISpiderLoader) BaseSettingsSettingsoverridden_settings) SignalManager)LogCounterHandlerconfigure_loggingget_scrapy_root_handlerinstall_scrapy_root_handlerlog_reactor_infolog_scrapy_info)build_from_crawler load_object)install_shutdown_handlers signal_names)install_reactoris_asyncio_reactor_installed#verify_installed_asyncio_event_loopverify_installed_reactor) GeneratorIterable) LogFormatter)SpiderLoaderProtocol)StatsCollector)RequestFingerprinterProtocol_Tc\rSrSrSSSjjrSSjrSSjr\SSj5rSSjr SSjr \SS j5r \ SS j5r SS jrSS jrSS jrSSjrSSjrSrg)Crawler7Nc[U[5(a [S5e[U[5(dUc [ U5nXlUR 5UlUR RUR5 UR5 [U5Ul [U5Ul X0lSUlSUlSUlSUlSUlSUlSUlSUlg)Nz5The spidercls argument must be a class, not an objectF) isinstancer ValueErrordictr spiderclscopysettingsupdate_settings_update_root_log_handlerraddonsrr _init_reactorcrawling_started extensionsstats logformatterrequest_fingerprinterspiderengine)selfr4r6 init_reactors U/root/1688_scrapy/alibaba-scraper/venv/lib/python3.13/site-packages/scrapy/crawler.py__init__Crawler.__init__8s i ( (TU U h % %)9)H'0"*--/  &&t}}5 %%'$0$6 &3D&9 #/# # 37,0 15JN"%) .2 cF[5b[UR5 ggN)rrr6rCs rEr8 Crawler._update_root_log_handlerWs " $ 0 ' 6 1rHc^URR(agURRUR5 [ URS5"U5Ul[ XRRS5S9m[RRT5 U4SjUl URRUR[R5 [ URS5nURU5Ul[#[ URS5U5UlURSnURSnUR&(aU(a [)X#5 OS S KJn U(a,[/U5 [15(aU(a [3U5 UR&(dU(a [55 [6R"U5UlURR;5 [=[?UR55n[@RCS S [DRF"U505 g) N STATS_CLASS LOG_LEVEL)levelcB>[RRT5$rJ)loggingroot removeHandler)handlersrE)Crawler._apply_settings..gs (B(B7(KrH LOG_FORMATTERREQUEST_FINGERPRINTER_CLASSTWISTED_REACTORASYNCIO_EVENT_LOOPrreactorz!Overridden settings: %(settings)sr6)$r6frozenr9 load_settingsrr>rgetrRrS addHandler_Crawler__remove_handlerrconnectengine_stopped from_crawlerr?rr@r:r"twisted.internetr]r%r#r$rrr=freezer3rloggerinfopprintpformat)rClf_cls reactor_class event_loopr]drUs @rE_apply_settingsCrawler._apply_settings\s ==    !!$--0 }!=>tD #D 0A0A+0NO (!L T22G4J4JK%01O%P"//5%7  &CD E & " "]]+<= --(<=    :4  $] 3+--*3J?     *77=  $T]]3 4 0:v~~a?P2Q rHc/6# UR(a [S5eUR(a [S5eS=UlUlUR"U0UD6UlUR 5 UR 5 UR5UlURRUR5v URR5v g![a2 SUlURbURR5v ef=f7f)NzCrawling already taking placez?Cannot run Crawler.crawl() more than once on the same instance.TF) r; RuntimeErrorr<_create_spiderrArpr8_create_enginerB open_spiderstart ExceptioncloserCargskwargss rEcrawl Crawler.crawls ==>? ? ==Q )-,   --t>v>DK  "  ) ) +--/DK++))$++6 6++##% % !DM{{&kk''))   sAD BCDTR5$rJ)stop)_rCs rErV(Crawler._create_engine..s tyy{rHrrKs`rEruCrawler._create_enginest%:;;rHc## UR(a<SUlUR(de[URR5v gg7f)zgStarts a graceful stop of the crawler and returns a deferred that is fired when the crawler is stopped.FN)r;rBr rrKs rEr Crawler.stops; ==!DM;; ; 0 01 1 sAAc>UHn[X 5(dMUs $ grJ)r1)component_class components components rE_get_componentCrawler._get_components$$I)55  $rHcLURXRR5$)zReturn the run-time instance of an :ref:`add-on ` of the specified class or a subclass, or ``None`` if none is found. .. versionadded:: 2.12 )rr9rCclss rE get_addonCrawler.get_addons ""3 (:(:;;rHcUR(d [S5eURXRRRR 5$)aHReturn the run-time instance of a :ref:`downloader middleware ` of the specified class or a subclass, or ``None`` if none is found. .. versionadded:: 2.12 This method can only be called after the crawl engine has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. z_Crawler.get_downloader_middleware() can only be called after the crawl engine has been created.)rBrsr downloader middleware middlewaresrs rEget_downloader_middleware!Crawler.get_downloader_middlewaresF{{5 ""3 (>(>(I(I(U(UVVrHcUR(d [S5eURXRR5$)a7Return the run-time instance of an :ref:`extension ` of the specified class or a subclass, or ``None`` if none is found. .. versionadded:: 2.12 This method can only be called after the extension manager has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. zXCrawler.get_extension() can only be called after the extension manager has been created.)r=rsrrrs rE get_extensionCrawler.get_extensions:6 ""3(C(CDDrHcUR(d [S5eURXRRRR 5$)a8Return the run-time instance of a :ref:`item pipeline ` of the specified class or a subclass, or ``None`` if none is found. .. versionadded:: 2.12 This method can only be called after the crawl engine has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. zWCrawler.get_item_pipeline() can only be called after the crawl engine has been created.)rBrsrscraperitemprocrrs rEget_item_pipelineCrawler.get_item_pipelineF{{1 ""3 (;(;(D(D(P(PQQrHcUR(d [S5eURXRRRR 5$)a@Return the run-time instance of a :ref:`spider middleware ` of the specified class or a subclass, or ``None`` if none is found. .. versionadded:: 2.12 This method can only be called after the crawl engine has been created, e.g. at signals :signal:`engine_started` or :signal:`spider_opened`. z[Crawler.get_spider_middleware() can only be called after the crawl engine has been created.)rBrsrrspidermwrrs rEget_spider_middlewareCrawler.get_spider_middlewarerrH)__remove_handlerr:r<r9r;rBr=r?r@r6rrAr4r>NF)r4z type[Spider]r6 dict[str, Any] | Settings | NonerDbool)returnNone)r{rr|rr#Generator[Deferred[Any], Any, None])r{rr|rrr )rrrr)rtype[_T]rz Iterable[Any]r _T | None)rrrr)__name__ __module__ __qualname____firstlineno__rFr8rpr r}rtrur staticmethodrrrrrr__static_attributes__rHrEr.r.7s6:" 3333 3>7 - d,B<22!/< <W"E$R"RrHr.c\rSrSrSr\"SSS9r\SSj5rSSSjjr SS jr SS jr SS jr SS jr SS jr\SSj5rSrg) CrawlerRunneri a This is a convenient helper class that keeps track of, manages and runs crawlers inside an already setup :mod:`~twisted.internet.reactor`. The CrawlerRunner object must be instantiated with a :class:`~scrapy.settings.Settings` object. This class shouldn't be needed (since Scrapy is responsible of using it accordingly) unless writing scripts that manually handle the crawling process. See :ref:`run-from-script` for an example. cUR$rJ) _crawlersrKs rErVCrawlerRunner.sT^^rHzeSet of :class:`crawlers ` started by :meth:`crawl` and managed by this class.)doccURS5n[U5n[[U5 [ SUR UR 555$)z'Get SpiderLoader instance from settingsSPIDER_LOADER_CLASSr))r`rr rr from_settings frozencopy)r6cls_path loader_clss rE_get_spider_loader CrawlerRunner._get_spider_loadersM<< 56 * M:. "J$<$TRRT5 TRRT5 T=R[ TSS5(+-slU$)NrA)crawlersdiscardrrgetattr)resultrrorCs rE_done#CrawlerRunner._crawl.._doneZsI MM ! !' * LL  #  ! !(D)I%I I !MrH)rr,rr,)raddr}raddBoth)rCrr{r|rros`` @rErCrawlerRunner._crawlUsU '" MM4 *6 *    yyrHc[U[5(a [S5e[U[5(aU$UR U5$)a Return a :class:`~scrapy.crawler.Crawler` object. * If ``crawler_or_spidercls`` is a Crawler, it is returned as-is. * If ``crawler_or_spidercls`` is a Spider subclass, a new Crawler is constructed for it. * If ``crawler_or_spidercls`` is a string, this function finds a spider with this name in a Scrapy project (using spider loader), then creates a Crawler instance for it. r)r1r r2r._create_crawler)rCrs rErCrawlerRunner.create_crawlerbsM *F 3 3B  *G 4 4' '##$899rHc[U[5(aURRU5n[ XR 5$rJ)r1strrloadr.r6)rCr4s rErCrawlerRunner._create_crawlerxs4 i % %**// :Iy--00rHc[[UR5Vs/sHoR5PM sn5$s snf)zv Stops simultaneously all the crawling jobs taking place. Returns a deferred that is fired when they all have ended. )r listrr)rCcs rErCrawlerRunner.stop}s0 tDMM/BC/B!VVX/BCDDCs=c## UR(a+[UR5v UR(aM*gg7f)zm join() Returns a deferred that is fired when all managed :attr:`crawlers` have completed their executions. N)rr rKs rEjoinCrawlerRunner.joins(llt||, ,llls :AA)rrrr6r)r6rrr)rJ)r6r)rtype[Spider] | str | Crawlerr{rr|rrDeferred[None])rr.r{rr|rrr)rrrr.)r4zstr | type[Spider]rr.rz Deferred[Any]r)rrrr__doc__propertyrrrrFr}rrrrr rrrrHrErr s # 3H   &!5:!5!5 !5  !5F  :$@: :,1 E--rHrc^\rSrSrSrS S U4SjjjrS SjrS SjrSSjrSSSjjr SSjr SSS jjr S r U=r $)CrawlerProcessia\ A class to run multiple scrapy crawlers in a process simultaneously. This class extends :class:`~scrapy.crawler.CrawlerRunner` by adding support for starting a :mod:`~twisted.internet.reactor` and handling shutdown signals, like the keyboard interrupt command Ctrl-C. It also configures top-level logging. This utility should be a better fit than :class:`~scrapy.crawler.CrawlerRunner` if you aren't running another :mod:`~twisted.internet.reactor` within your application. The CrawlerProcess object must be instantiated with a :class:`~scrapy.settings.Settings` object. :param install_root_handler: whether to install root logging handler (default: True) This class shouldn't be needed (since Scrapy is responsible of using it accordingly) unless writing scripts that manually handle the crawling process. See :ref:`run-from-script` for an example. c>[TU]U5 [URU5 [ UR5 SUlgr)superrFrr6r_initialized_reactor)rCr6install_root_handler __class__s rErFCrawlerProcess.__init__s5 "$--)=> &*/!rHcSSKJn [UR5 [Un[ R SSU05 URUR5 g)Nrr\zDReceived %(signame)s, shutting down gracefully. Send again to force signame) rfr]r _signal_killr!rhricallFromThread_graceful_stop_reactorrCsignumrr]rs rE_signal_shutdownCrawlerProcess._signal_shutdownsJ,!$"3"34v& R    t::;rHcSSKJn [[R5 [ Un[ RSSU05 URUR5 g)Nrr\z4Received %(signame)s twice, forcing unclean shutdownr) rfr]r signalSIG_IGNr!rhrir _stop_reactorrs rErCrawlerProcess._signal_killsH,!&..1v& BYPWDX  t112rHc[U[5(aURRU5nUR(+nSUl[ XR US9$)NT)rD)r1rrrrr.r6)rCr4rDs rErCrawlerProcess._create_crawlersL i % %**// :I444 $(!y--lKKrHcSSKJn U(a=UR5nUR(agUR UR 5 [ URS5n[XPUS9nUR5 UR5nURURRS5S9 URSSUR5 U(a"URS S [UR 5 UR#US 9 g) a This method starts a :mod:`~twisted.internet.reactor`, adjusts its pool size to :setting:`REACTOR_THREADPOOL_MAXSIZE`, and installs a DNS cache based on :setting:`DNSCACHE_ENABLED` and :setting:`DNSCACHE_SIZE`. If ``stop_after_crawl`` is True, the reactor will be stopped after all crawlers have finished, using :meth:`join`. :param bool stop_after_crawl: stop or not the reactor when all crawlers have finished :param bool install_signal_handlers: whether to install the OS signal handlers from Twisted and Scrapy (default: True) rr\N DNS_RESOLVERREACTOR_THREADPOOL_MAXSIZE) maxthreadsbeforeshutdownafterstartup)installSignalHandlers)rfr]rcalledrrrr6rinstall_on_reactor getThreadPooladjustPoolsizegetintaddSystemEventTriggerrr rrun)rCstop_after_crawlinstall_signal_handlersr]roresolver_classresolvertps rErwCrawlerProcess.starts" -  Axx IId(( )$T]]>%BC&nGL##%  " " $ T]]%9%9:V%WX%%h DIIF "  ) )$=t?T?T   *A BrHc\UR5nURUR5 U$rJ)rrr)rCros rEr%CrawlerProcess._graceful_stop_reactors$ IIK $$$%rHcSSKJn [R"[5 UR 5 SSS5 g!,(df  g=f)Nrr\)rfr] contextlibsuppressrsr)rCrr]s rErCrawlerProcess._stop_reactors*,  . LLN/ . .s ; A )r)NT)r6rrr)rintrrrr)r4ztype[Spider] | strrr.)TT)rrrrrrrrJ)rrrr)rrrrrrFrrrrwrrr __classcell__)rs@rErrst26:%)020#00 <3LNR&C $&CFJ&C &CP rHr)G __future__rr#rRrjrtypingrrrrtwisted.internet.deferrr r r zope.interface.verifyr scrapyr r scrapy.addonsrscrapy.core.enginerscrapy.extensionrscrapy.interfacesrscrapy.settingsrrrscrapy.signalmanagerrscrapy.utils.logrrrrrrscrapy.utils.miscrrscrapy.utils.ossignalr r!scrapy.utils.reactorr"r#r$r%collections.abcr&r'scrapy.logformatterr(scrapy.spiderloaderr)scrapy.statscollectorsr*scrapy.utils.requestr+ getLoggerrrhr,r.rrrrHrEr=s" 44 ."&.-+GG.>I3085A   8 $ T]QRQRhC-C-Lq]qrH