!h: |%SrSSKJr SSKrSSKJrJr SSKJr SSK J r SSK J r J r SSKJr SS KJrJr SS KJr SS KJr SS KJr SS KJr \(aSSKJr SSKJr SSKJ r Sr!S\"S'Sr#Sr$Sr%Sr&Sr'Sr(Sr)Sr*Sr+"SS5r,"SS \,5r-"S!S"\,5r."S#S$\,5r/"S%S&\,5r0"S'S(\,5r1"S)S*\,5r2"S+S,\,5r3"S-S.\,5r4"S/S0\.5r5\-\.\/\0\1\2\3\4\54 Vs0sHnURlU_M snr7S1\"S2'\.\7S3'S7S8S4jjr8"S5S6\5r9gs snf)9z` RefererMiddleware: populates Request referer field, based on the Response which originated it. ) annotationsN) TYPE_CHECKINGcast)urlparse)safe_url_string)Spidersignals) NotConfigured)RequestResponse)BaseSpiderMiddleware) load_object) to_unicode strip_url)Self)Crawler) BaseSettings)aboutblobdata filesystemtuple[str, ...] LOCAL_SCHEMESz no-referrerzno-referrer-when-downgradez same-originoriginz strict-originzorigin-when-cross-originzstrict-origin-when-cross-originz unsafe-urlzscrapy-defaultcx\rSrSr%\rS\S'S\S'SSjrSSjrSSjr SSS jjr SS jr SS jr SS jr S rg)ReferrerPolicy0rNOREFERRER_SCHEMESstrnamec[eN)NotImplementedErrorself response_url request_urls g/root/1688_scrapy/alibaba-scraper/venv/lib/python3.13/site-packages/scrapy/spidermiddlewares/referer.pyreferrerReferrerPolicy.referrer4s!!cl[U5RUR;aURU5$gr#)rschemerrr&urls r)stripped_referrer ReferrerPolicy.stripped_referrer7s, C=  t'>'> >>>#& &r,cl[U5RUR;aURU5$gr#)rr.rrr/s r)origin_referrerReferrerPolicy.origin_referrer<s, C=  t'>'> >;;s# #r,c,U(dg[USSSUS9$)a_ https://www.w3.org/TR/referrer-policy/#strip-url If url is null, return no referrer. If url's scheme is a local scheme, then return no referrer. Set url's username to the empty string. Set url's password to null. Set url's fragment to null. If the origin-only flag is true, then: Set url's path to null. Set url's query to null. Return url. NT)strip_credentialsstrip_fragmentstrip_default_port origin_onlyr)r&r0r:s r)rReferrerPolicy.strip_urlAs& "##   r,c"URUSS9$)zLReturn serialized origin (scheme, host, path) for a request or response URL.T)r:rr/s r)rReferrerPolicy.originYs~~ct~44r,c\[U5nURS;agURU5$)N)rF)rr. tls_protected)r&r0 parsed_urls r)potentially_trustworthy&ReferrerPolicy.potentially_trustworthy]s-c]    )!!#&&r,c2[U5RS;$)N)httpsftps)rr.r/s r)r?ReferrerPolicy.tls_protectedds}##'888r,Nr'r r(r return str | None)r0r rIrJF)r0r r:boolrIrJ)r0r rIrL)__name__ __module__ __qualname____firstlineno__rr__annotations__r*r1r4rrrAr?__static_attributes__rGr,r)rr0s5*77 I"   05'9r,rc2\rSrSr%Sr\rS\S'SSjrSr g) NoReferrerPolicyha https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer The simplest policy is "no-referrer", which specifies that no referrer information is to be sent along with requests made from a particular request client to any origin. The header will be omitted entirely. r r!cgr#rGr%s r)r*NoReferrerPolicy.referrerssr,rGNrH) rMrNrOrP__doc__POLICY_NO_REFERRERr!rQr*rRrGr,r)rTrThs#D#"r,rTc2\rSrSr%Sr\rS\S'SSjrSr g) NoReferrerWhenDowngradePolicywa3 https://www.w3.org/TR/referrer-policy/#referrer-policy-no-referrer-when-downgrade The "no-referrer-when-downgrade" policy sends a full URL along with requests from a TLS-protected environment settings object to a potentially trustworthy URL, and requests from clients which are not TLS-protected to any origin. Requests from TLS-protected clients to non-potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. This is a user agent's default behavior, if no policy is otherwise specified. r r!c~URU5(aURU5(aURU5$gr#)r?r1r%s r)r*&NoReferrerWhenDowngradePolicy.referrers7!!,//43E3Ek3R3R)),7 7r,rGNrH) rMrNrOrPrX!POLICY_NO_REFERRER_WHEN_DOWNGRADEr!rQr*rRrGr,r)r[r[ws 2D#1r,r[c2\rSrSr%Sr\rS\S'SSjrSr g) SameOriginPolicyau https://www.w3.org/TR/referrer-policy/#referrer-policy-same-origin The "same-origin" policy specifies that a full URL, stripped for use as a referrer, is sent as referrer information when making same-origin requests from a particular request client. Cross-origin requests, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. r r!cnURU5URU5:XaURU5$gr#rr1r%s r)r*SameOriginPolicy.referrers0 ;;| $ K(@ @)),7 7r,rGNrH) rMrNrOrPrXPOLICY_SAME_ORIGINr!rQr*rRrGr,r)raras#D#"r,rac2\rSrSr%Sr\rS\S'SSjrSr g) OriginPolicya& https://www.w3.org/TR/referrer-policy/#referrer-policy-origin The "origin" policy specifies that only the ASCII serialization of the origin of the request client is sent as referrer information when making both same-origin requests and cross-origin requests from a particular request client. r r!c$URU5$r#)r4r%s r)r*OriginPolicy.referrers##L11r,rGNrH) rMrNrOrPrX POLICY_ORIGINr!rQr*rRrGr,r)rhrhsD#2r,rhc2\rSrSr%Sr\rS\S'SSjrSr g) StrictOriginPolicya  https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin The "strict-origin" policy sends the ASCII serialization of the origin of the request client when making requests: - from a TLS-protected environment settings object to a potentially trustworthy URL, and - from non-TLS-protected environment settings objects to any origin. Requests from TLS-protected request clients to non- potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. r r!cURU5(aURU5(dURU5(dURU5$gr#)r?rAr4r%s r)r*StrictOriginPolicy.referrersI   | , ,,,[99##L11'' 5 5r,rGNrH) rMrNrOrPrXPOLICY_STRICT_ORIGINr!rQr*rRrGr,r)rnrns %D#$r,rnc2\rSrSr%Sr\rS\S'SSjrSr g) OriginWhenCrossOriginPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-origin-when-cross-origin The "origin-when-cross-origin" policy specifies that a full URL, stripped for use as a referrer, is sent as referrer information when making same-origin requests from a particular request client, and only the ASCII serialization of the origin of the request client is sent as referrer information when making cross-origin requests from a particular request client. r r!crURU5nX0RU5:XaURU5$U$r#rdr&r'r(rs r)r*$OriginWhenCrossOriginPolicy.referrers5\* [[- -)),7 7 r,rGNrH) rMrNrOrPrXPOLICY_ORIGIN_WHEN_CROSS_ORIGINr!rQr*rRrGr,r)rtrts 0D#/r,rtc2\rSrSr%Sr\rS\S'SSjrSr g) !StrictOriginWhenCrossOriginPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-strict-origin-when-cross-origin The "strict-origin-when-cross-origin" policy specifies that a full URL, stripped for use as a referrer, is sent as referrer information when making same-origin requests from a particular request client, and only the ASCII serialization of the origin of the request client when making cross-origin requests: - from a TLS-protected environment settings object to a potentially trustworthy URL, and - from non-TLS-protected environment settings objects to any origin. Requests from TLS-protected clients to non- potentially trustworthy URLs, on the other hand, will contain no referrer information. A Referer HTTP header will not be sent. r r!cURU5nX0RU5:XaURU5$URU5(aURU5(dURU5(dUR U5$gr#)rr1r?rAr4rws r)r**StrictOriginWhenCrossOriginPolicy.referrersw\* [[- -)),7 7   | , ,,,[99##L11'' 5 5r,rGNrH) rMrNrOrPrX&POLICY_STRICT_ORIGIN_WHEN_CROSS_ORIGINr!rQr*rRrGr,r)r{r{s"7D#6 r,r{c2\rSrSr%Sr\rS\S'SSjrSr g) UnsafeUrlPolicya https://www.w3.org/TR/referrer-policy/#referrer-policy-unsafe-url The "unsafe-url" policy specifies that a full URL, stripped for use as a referrer, is sent along with both cross-origin requests and same-origin requests made from a particular request client. Note: The policy's name doesn't lie; it is unsafe. This policy will leak origins and paths from TLS-protected resources to insecure origins. Carefully consider the impact of setting such a policy for potentially sensitive documents. r r!c$URU5$r#)r1r%s r)r*UnsafeUrlPolicy.referrers%%l33r,rGNrH) rMrNrOrPrXPOLICY_UNSAFE_URLr!rQr*rRrGr,r)rrs "D#!4r,rcD\rSrSr%Sr/\QSPSP7rS\S'\r S\S'S r g ) DefaultReferrerPolicyiz A variant of "no-referrer-when-downgrade", with the addition that "Referer" is not sent if the parent request was using ``file://`` or ``s3://`` scheme. files3rrr r!rGN) rMrNrOrPrXrrrQPOLICY_SCRAPY_DEFAULTr!rRrGr,r)rrs. +IM*H6*H4*HH%D#%r,rzdict[str, type[ReferrerPolicy]]_policy_classesc[[[[U55$![a UR 5R S5Vs/sHo"R5PM Os snfnnUSSS2HnU[;dM[Us s$ SU<3nU(d [U5e[R"U[5 gf=f)z Expect a string for the path to the policy class, otherwise try to interpret the string as a standard value from https://www.w3.org/TR/referrer-policy/#referrer-policies ,NzCould not load referrer policy ) rtyperr ValueErrorlowersplitstripr RuntimeErrorwarningswarnRuntimeWarning)policy warning_onlytokentokensmsgs r)_load_policy_classr1s D(+f*=>>  -3\\^-A-A#-FG-FE++--FGGDbD\E'&u--"0z:s# # c>* s'"+C  A'&C  C 5C  C c`\rSrSrS S Sjjr\S Sj5rS SjrS SjrSSjr Sr g)RefererMiddlewareiINct[UlUb*[URS55nU(deX lgg)NREFERRER_POLICY)rdefault_policyrget)r&settingssettings_policys r)__init__RefererMiddleware.__init__Js74I  0>O1PQO" "?"1  r,cURRS5(d[eU"UR5nURR UR [R S9 U$)NREFERER_ENABLED)signal)rgetboolr r connectrequest_scheduled)clscrawlermws r) from_crawlerRefererMiddleware.from_crawlerQsZ''(9::  !! "  4 4W=V=VW r,cNURRS5nUcM[U[5(a8URRS5nUb[ UR S55nUcUR5$[USS9nU(aU"5$UR5$)a Determine Referrer-Policy to use from a parent Response (or URL), and a Request to be sent. - if a valid policy is set in Request meta, it is used. - if the policy is set in meta but is wrong (e.g. a typo error), the policy from settings is used - if the policy is not set in Request meta, but there is a Referrer-policy header in the parent response, it is used if valid - otherwise, the policy from settings is used. referrer_policyzReferrer-Policylatin1T)r) metar isinstancer headersrdecoderr)r& resp_or_urlrequest policy_name policy_headerrs r)rRefererMiddleware.policy\sll&&'89  :k8#D#D'//334EFM(()=)=h)GH  &&( ( 4@su6!4!4!66r,cUcU$URX!5RURUR5nUbURR SU5 U$)NReferer)rr*r0r setdefault)r&rresponser*s r)get_processed_request'RefererMiddleware.get_processed_requesttsQ  N;;x1::8<<U   OO & &y( ;r,cxURRS/5nU(aURRS5nUbx[U5nUR XQ5R XQR 5nXdRS5:wa.UcURRS5 gXaRS'gggg)N redirect_urlsrr) rrrrrr*r0rpop)r&rspiderredirected_urlsrequest_referrer parent_urlpolicy_referrers r)r#RefererMiddleware.request_scheduleds!,,**?B? &229= + --=> "&++j"B"K"K ##&=&=h&GG&.++I65D 2 H, r,)rr#)rzBaseSettings | None)rrrIr)rzResponse | strrr rIr)rr rzResponse | NonerIzRequest | None)rr rrrINone) rMrNrOrPr classmethodrrrrrRrGr,r)rrIsC270  *9   Er,rrK)rr rrLrIztype[ReferrerPolicy] | None):rX __future__rrtypingrr urllib.parser w3lib.urlrscrapyrr scrapy.exceptionsr scrapy.httpr r scrapy.spidermiddlewares.baser scrapy.utils.miscrscrapy.utils.pythonrscrapy.utils.urlrtyping_extensionsrscrapy.crawlerrscrapy.settingsrrrQrYr_rfrlrrryrrrrrTr[rarhrnrtr{rrr!rrr)ps0r)rs #&!%"+)>)*&&&," #$@!" &"<)J& (5959p ~ N.~& 2> 2 2.*@4n4(&9& %#)  4 FFAI  40  4', # 0JE,JEW 4s<D9