!h8!%SrSSKJr SSKrSSKrSSKrSSKJrJrJ r SSK J r SSK J r SSKJr SSKJr SS KJrJr SS KJr SS KJr SS KJr SS KJrJr \(aSSKJr SSK J!r! SSK"J#r# \ "5r$S\%S'SSS.S!Sjjr&"SS\ 5r'"SS5r(S"Sjr)S#Sjr*S$Sjr+SS.S%Sjjr,S&Sjr-S'S jr.g)(zT This module provides some useful functions for working with scrapy.Request objects ) annotationsN) TYPE_CHECKINGAnyProtocol) urlunparse)WeakKeyDictionary)basic_auth_header)canonicalize_url)RequestSpider)ScrapyDeprecationWarning)urlparse_cached) load_object)to_bytes to_unicode)Iterable)Self)CrawlerzNWeakKeyDictionary[Request, dict[tuple[tuple[bytes, ...] | None, bool], bytes]]_fingerprint_cacheF)include_headerskeep_fragmentscSnU(a[S[U555n[RU05nX24nXT;Ga0nU(adUH^nXpR;dMURR U5Vs/sHnUR 5PM snXgR 5'M` [UR5[URUS9UR=(d SR 5US.n [R"U SS9n [R"U R!55R#5XE'XE$s snf)a Return the request fingerprint. The request fingerprint is a hash that uniquely identifies the resource the request points to. For example, take the following two urls: ``http://www.example.com/query?id=111&cat=222``, ``http://www.example.com/query?cat=222&id=111``. Even though those are two different URLs both point to the same resource and are equivalent (i.e. they should return the same response). Another example are cookies used to store session ids. Suppose the following page is only accessible to authenticated users: ``http://www.example.com/members/offers.html``. Lots of sites use a cookie to store the session id, which adds a random component to the HTTP Request and thus should be ignored when calculating the fingerprint. For this reason, request headers are ignored by default when calculating the fingerprint. If you want to include specific headers use the include_headers argument, which is a list of Request headers to include. Also, servers usually ignore fragments in urls when handling requests, so they are also ignored by default when calculating the fingerprint. If you want to include them, set the keep_fragments argument to True (for instance when handling requests with a headless browser). Nc3T# UHn[UR55v M g7fN)rlower).0hs [/root/1688_scrapy/alibaba-scraper/venv/lib/python3.13/site-packages/scrapy/utils/request.py fingerprint..Js"* )@AHQWWY  )@s&()r)methodurlbodyheadersT) sort_keys)tuplesortedr setdefaultr%getlisthexrr"r r#r$jsondumpshashlibsha1encodedigest) requestrrprocessed_include_headerscache cache_keyr%header header_valuefingerprint_datafingerprint_jsons r fingerprintr:&s2D;?$)* )/)@* % !  ) )'2 6E*;I)+ $3__,-4OO,C,CF,K-,KL%((*,K-GJJL)4!0#GKKO\\(S--/    ::&6$G"<<  # # % &(   -sE c\rSrSrSSjrSrg)RequestFingerprinterProtocolgcgrselfr2s rr:(RequestFingerprinterProtocol.fingerprinthscr!r?Nr2r returnbytes)__name__ __module__ __qualname____firstlineno__r:__static_attributes__r?r!rr<r<gs9r!r<cD\rSrSrSr\SSj5rS S SjjrS SjrSr g) RequestFingerprinterkaMDefault fingerprinter. It takes into account a canonical version (:func:`w3lib.url.canonicalize_url`) of :attr:`request.url ` and the values of :attr:`request.method ` and :attr:`request.body `. It then generates an `SHA1 `_ hash. cU"U5$rr?)clscrawlers r from_crawler!RequestFingerprinter.from_crawlervs 7|r!NcU(aURRS5nOSnUS:waSn[R"U[SS9 [ Ulg)N$REQUEST_FINGERPRINTER_IMPLEMENTATIONSENTINELzq'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy.category stacklevel)settingsgetwarningswarnr r: _fingerprint)rArPimplementationmessages r__init__RequestFingerprinter.__init__zsT $--116N(N Z 'D  MM',DQR S'r!c$URU5$rr^r@s rr: RequestFingerprinter.fingerprints  ))r!rd)rPrrDrr)rPzCrawler | NonerC) rFrGrHrI__doc__ classmethodrQrar:rJr?r!rrLrLks%( *r!rLch[R"S[SS9 [X5URS'g)zAuthenticate the given request (in place) using the HTTP basic access authentication mechanism (RFC 2617) and the given username and password zbThe request_authenticate function is deprecated and will be removed in a future version of Scrapy.rVrW AuthorizationN)r\r]r r r%)r2usernamepasswords rrequest_authenticaterls. MMl) (9'LGOOO$r!c[U5n[SSUR=(d SURURS45n[ UR 5S-[ U5-S-nUS[ UR=(d S5-S-- nUR(aX0RR5S-- nUS- nX0R- nU$)zReturn the raw HTTP representation (as bytes) of the given request. This is provided only for reference since it's not the actual stream of bytes that will be send when performing the request (that's controlled by Twisted). / s HTTP/1.1 sHost: r!s ) rrpathparamsqueryrr"hostnamer% to_stringr$)r2parsedrqss rrequest_httpreprrxs W %F r2v{{1c6==&,,PRS TD 4'(4.8;KKAXfoo45 5 ??A __ & & (7 22LAA Hr!cVURRS5nUcU$[USS9$)z0Return Referer HTTP header suitable for logging.Refererreplace)errors)r%r[r)r2referrers r referer_strr~s.""9-H hy 11r!)spiderctSU;a[US5O[nUR5VVs0sHup4X2R;dMX4_M nnnUR S5(aU(a[ XS5US'UR S5(aU(a[ XS5US'U"S0UD6$s snnf)zCreate a :class:`~scrapy.Request` object from a dict. If a spider is given, it will try to resolve the callbacks looking at the spider for methods with the same name. _classcallbackerrbackr?)rr items attributesr[ _get_method)dr request_clskeyvaluekwargss rrequest_from_dictrs >F]Qx[!9PWK+,779 V9ZS?U?U8Ujcj9F VuuZV(:?zuuYF') =y    Ws B4B4cp[U5n[X5$![a [SU<SU35ef=f)z%Helper function for request_from_dictzMethod z not found in: )strgetattrAttributeError ValueError)objnames rrrsE t9DAs!! A74(/#?@@As 5c URnUR(aSURRS5S3OSnSRSURR 555nUR nSnUR(a[UR[5(a7SRSURR 555nS US3nOG[UR[5(a(SRS UR55nS US3nS USUSUSUSU3 R5nSRUR55$) z Converts a :class:`~scrapy.Request` object to a curl command. :param :class:`~scrapy.Request`: Request object to be converted :return: string containing the curl command z --data-raw 'zutf-8'rn c3v# UH/upSUR5SUSR5S3v M1 g7f)z-H 'z: rrN)decoderkvs rr"request_to_curl..s5:Q$!$qxxzl"QqT[[]O1-:Qs79z; c34# UHupUSU3v M g7f=Nr?rs rrrsN6Mda!AaSz6Msz --cookie 'c 3# UHLn[[UR555S[[UR5553v MN g7fr)nextiterkeysvalues)rcs rrrsC(AQVVX'($tAHHJ/?*@)AB(sAAzcurl -X ) r"r$rjoinr%rr#cookies isinstancedictliststripsplit)r2r"datar%r#rcookiecurl_cmds rrequest_to_curlrs5^^F=D\\\',,--g67q 9rDhh:A//:O:O:QG ++CG goot , ,YYNgoo6K6K6MNNF"6(!,G  . .YY F#6(!,G&3%qay'CIIKH 88HNN$ %%r!)r2r rzIterable[bytes | str] | NonerboolrDrE)r2r rjrrkrrDNonerC)r2r rDz str | None)rzdict[str, Any]rz Spider | NonerDr )rrrrrDr)r2r rDr)/rf __future__rr.r,r\typingrrr urllib.parserweakrefr w3lib.httpr w3lib.urlr scrapyr r scrapy.exceptionsr scrapy.utils.httpobjrscrapy.utils.miscrscrapy.utils.pythonrrcollections.abcrtyping_extensionsrscrapy.crawlerrr__annotations__r:r<rLrlrxr~rrrr?r!rrs # //#%(&"60)4('& 59 > >2> >  >B:8: * *F M  M M M M  "2EI !A&r!