h: SrSSKJr SSKrSSKrSSKrSSKrSSKJr SSK J r SSK J r SSK JrJrJrJr SSKJrJrJr SS KJr SS KJr SS KJrJrJr SS KJ r SS K!J"r" SSK#J$r$J%r% SSK&J'r'J(r(J)r)J*r*J+r+ SSK,J-r- SSK.J/r/J0r0 SSK1J2r2 SSK3J4r4J5r5 SSK6J7r7 SSK8J9r9J:r: SSK;JJ?r?J@r@ SSKAJBrBJCrCJDrDJErEJFrFJGrG \(aSSKHJIrJ SSKKJLrM SSKNJOrP O\QrP\QrM\QrJSr"SS\9R5rS"SS 5rTS!rUS^S"jrVSS#K;JWrWJXrXJOrOJYrYJZrZJ[r[J\r\J]r]J^r^J_r_J`r` SS$KaJbrb SS%KcJdrd SS&KeJfrfJgrgJhrh S'rj"S(S)5rk\"\F5"S*S+55rl\4"\"S,S-SS5\5"\k\"S,S-SS5\lS.9RS/5S0\kR\kR5 \"\F5"S1S255rp\"\(5"S3S455rq\"\F5"S5S655rr\"\E5"S7S855rs"S9S:\R5ru"S;S<5rv"S=S>5rw"S?S@5rx\"\D5"SASB55ry\"\C5"SCSD\x55rz\"\C5"SESF\x55r{"SGSH\P5r|\"SISJ\J55r}"SKSL\M5r~\"\C5"SMSN55r"SOSP\2"\G55r"SQSR\2"\)55r\"\C5"SSST55r\@GRr\"/SUQ5r\"\C5"SVSW55r"SXSY\5r"SZS[\GR5rS_S\jr/S]Qrg!\ia SrdGNf=f)`z HTTP client. ) annotationsN) dataclass)wraps) CookieJar) TYPE_CHECKINGHashableIterableOptional) urldefragurljoin urlunparse) implementer)Version)deferprotocoltask) isIPv6Address)Deferred)HostnameEndpoint wrapClientTLS)IAddressIOpenSSLContextFactory IProtocol IReactorTimeIStreamClientEndpoint)Logger) nativeString networkString)proxyForInterface)deprecatedModuleAttributegetDeprecationWarningString)Failure)errorhttp)_ensureValidMethod_ensureValidURI)Headers _nameEncoder)UNKNOWN_LENGTHIAgentIAgentEndpointFactory IBodyProducerIPolicyForHTTPS IResponse) EmailMessage) HTTPResponse)RequestcZ[[SU555nURS5$)Nc3B# UHoRS5v M g7f)charmapN)decode).0ps Y/root/1688_scrapy/alibaba-scraper/venv/lib/python3.13/site-packages/twisted/web/client.py urlunparse..HsBEqxx 22Esr4) _urlunparsetupleencode)partsresults r8r r Gs& BEBB CF == ##c\rSrSrSrSrg)PartialDownloadErrorLz Page was only partially downloaded, we got disconnected in middle. @ivar response: All of the response body which was downloaded. N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__rDr@r8rBrBLsr@rBcH\rSrSrSrSr\S Sj5rSr\ S5r Sr g) URITzX A URI object. @see: U{https://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21} c XlX lURS5UlX@lXPlX`lXplXlg)a @type scheme: L{bytes} @param scheme: URI scheme specifier. @type netloc: L{bytes} @param netloc: Network location component. @type host: L{bytes} @param host: Host name. For IPv6 address literals the brackets are stripped. @type port: L{int} @param port: Port number. @type path: L{bytes} @param path: Hierarchical path. @type params: L{bytes} @param params: Parameters for last path segment. @type query: L{bytes} @param query: Query string. @type fragment: L{bytes} @param fragment: Fragment identifier. s[]N) schemenetlocstriphostportpathparamsqueryfragment) selfrOrPrRrSrTrUrVrWs r8__init__ URI.__init__[s76  JJu%      r@Nc UR5n[R"U5up4pVpxUc US;aSnOSnSU;a!URSS5up[ U 5n OXBpU"X4XXVXx5$![ a XBpNf=f)a Parse the given URI into a L{URI}. @type uri: C{bytes} @param uri: URI to parse. @type defaultPort: C{int} or L{None} @param defaultPort: An alternate value to use as the port if the URI does not include one. @rtype: L{URI} @return: Parsed URI instance. wsshttpsP:)rQr$urlparsersplitint ValueError) clsuri defaultPortrOrPrTrUrVrWrRrSs r8 fromBytes URI.fromBytessiik8< c8J5e  ++!  6>tQ/JD 14y $64tUMM  1#d 1s A-- A<;A<c[URURURURUR UR 45$)z{ Assemble the individual parts of the I{URI} into a fully formed I{URI}. @rtype: C{bytes} @return: A fully formed I{URI}. )r rOrPrTrUrVrWrXs r8toBytes URI.toBytess@         r@ct[SSURURURS45nUS:XaSnU$)z The absolute I{URI} path including I{URI} parameters, query string and fragment identifier. @see: U{https://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-21#section-5.3} @return: The absolute path in original form. @rtype: L{bytes} r@/)r rTrUrV)rXrTs r8 originFormURI.originForms83TYY TZZMN 3;D r@)rWrRrPrUrTrSrVrON) rErFrGrHrIrY classmethodrjrnpropertyrrrJrDr@r8rLrLTs= "!HNNB $r@rLcx[U5up[[X55up[USU=(d U-5$)a Construct a full ("absolute") URL by combining a "base URL" with another URL. Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL. Additionally, the fragment identifier is preserved according to the HTTP 1.1 bis draft. @type base: C{bytes} @param base: Base URL. @type url: C{bytes} @param url: URL to combine with C{base}. @return: An absolute URL resulting from the combination of C{base} and C{url}. @see: L{urllib.parse.urljoin()} @see: U{https://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-22#section-7.1.2} #)r r )baseurlbaseFragurlFrags r8_urljoinr}s7.t_NDWT/0LC 3 384 55r@c[R[UR555nU"U/UQ70UD6nSSKJn UR S:XaKSSKJn UcUR5nUR[UR5URXb5 U$UR[UR5URU5 U$)a Create and connect an HTTP page getting factory. Any additional positional or keyword arguments are used when calling C{factoryFactory}. @param factoryFactory: Factory factory that is called with C{url}, C{args} and C{kwargs} to produce the getter @param contextFactory: Context factory to use when creating a secure connection, defaulting to L{None} @return: The factory created by C{factoryFactory} rreactorr^)ssl)rLrjr&rQtwisted.internetrrOrClientContextFactory connectSSLrrRrS connectTCP) rzfactoryFactorycontextFactoryargskwargsrhfactoryrrs r8_makeGetterFactoryrs -- 4 5CS24262G( zzX(  ! 557N<1388WU N <1388WE Nr@) HTTP11ClientProtocolPotentialDataLossr1RequestGenerationFailedRequestNotSentRequestTransmissionFailedResponse ResponseDoneResponseFailedResponseNeverReceived_WrapperException)SchemeNotSupported)SSL)CertificateOptionsoptionsForClientTLS platformTrustc:[c[U5S5nU$U$)a] The decorated method requires pyOpenSSL to be present, or it raises L{NotImplementedError}. @param decoratee: A function which requires pyOpenSSL. @type decoratee: L{callable} @return: A function which raises L{NotImplementedError} if pyOpenSSL is not installed; otherwise, if it is installed, simply return C{decoratee}. @rtype: L{callable} c[S5e)z pyOpenSSL is not available. @param a: The positional arguments for C{decoratee}. @param kw: The keyword arguments for C{decoratee}. @raise NotImplementedError: Always. zSSL support unavailable)NotImplementedError)akws r8raiseNotImplemented(_requireSSL..raiseNotImplemented-s&&?@ @r@)rr) decorateers r8 _requireSSLrs0 { y  A  A#" r@c.\rSrSrSrSr\S5rSrg)WebClientContextFactoryi>a This class is deprecated. Please simply use L{Agent} as-is, or if you want to customize something, use L{BrowserLikePolicyForHTTPS}. A L{WebClientContextFactory} is an HTTPS policy which totally ignores the hostname and port. It performs basic certificate verification, however the lack of validation of service identity (e.g. hostname validation) means it is still vulnerable to man-in-the-middle attacks. Don't use it any more. cB[[R[5S9$)z Return a L{CertificateOptions}. @param hostname: ignored @param port: ignored @return: A new CertificateOptions instance. @rtype: L{CertificateOptions} )method trustRoot)rr SSLv23_METHODrrXhostnamerSs r8_getCertificateOptions.WebClientContextFactory._getCertificateOptionsIs"):):moVVr@c@URX5R5$)z Return an L{OpenSSL.SSL.Context}. @param hostname: ignored @param port: ignored @return: A new SSL context. @rtype: L{OpenSSL.SSL.Context} )r getContextrs r8r"WebClientContextFactory.getContextVs**8:EEGGr@rDN) rErFrGrHrIrrrrJrDr@r8rr>s# W H Hr@rc2\rSrSrSrSSjr\S5rSrg)BrowserLikePolicyForHTTPSidz) SSL connection creator for web clients. NcXlgrt _trustRoot)rXrs r8rY"BrowserLikePolicyForHTTPS.__init__js#r@cH[URS5URS9$)a Create a L{client connection creator } for a given network location. @param hostname: The hostname part of the URI. @type hostname: L{bytes} @param port: The port part of the URI. @type port: L{int} @return: a connection creator with appropriate verification restrictions set @rtype: L{client connection creator } ascii)r)rr5rrs r8creatorForNetloc*BrowserLikePolicyForHTTPS.creatorForNetlocms$#8??7#;tWWr@rrt) rErFrGrHrIrYrrrJrDr@r8rrds"$XXr@rTwisted) replacementz; rbc(\rSrSrSrSSjrSrSrg)HostnameCachingHTTPSPolicyia IPolicyForHTTPS that wraps a L{IPolicyForHTTPS} and caches the created L{IOpenSSLClientConnectionCreator}. This policy will cache up to C{cacheSize} L{client connection creators } for reuse in subsequent requests to the same hostname. @ivar _policyForHTTPS: See C{policyforHTTPS} parameter of L{__init__}. @ivar _cache: A cache associating hostnames to their L{client connection creators }. @type _cache: L{collections.OrderedDict} @ivar _cacheSize: See C{cacheSize} parameter of L{__init__}. @since: Twisted 19.2.0 cPXl[R"5UlX lg)z @param policyforHTTPS: The IPolicyForHTTPS to wrap. @type policyforHTTPS: L{IPolicyForHTTPS} @param cacheSize: The maximum size of the hostname cache. @type cacheSize: L{int} N)_policyForHTTPS collections OrderedDict_cache _cacheSize)rXpolicyforHTTPS cacheSizes r8rY#HostnameCachingHTTPSPolicy.__init__s .!--/ #r@cJURS5nURRU5nX@RU'[ UR5UR:aURRSS9 U$![a URR X5nNsf=f)a Create a L{client connection creator } for a given network location and cache it for future use. @param hostname: The hostname part of the URI. @type hostname: L{bytes} @param port: The port part of the URI. @type port: L{int} @return: a connection creator with appropriate verification restrictions set @rtype: L{client connection creator } rF)last) r5rpopKeyErrorrrlenrpopitem)rXrrSrRcreators r8r+HostnameCachingHTTPSPolicy.creatorForNetlocs"w' Lkkood+G$ D t{{ doo - KK  U  + L**;;HKG LsA::%B"!B")rrrN)rErFrGrHrIrYrrJrDr@r8rrs* $r@rc$\rSrSrSrSrSrSrg)_ContextFactoryWithContextiz A L{_ContextFactoryWithContext} is like a L{twisted.internet.ssl.ContextFactory} with a pre-created context. @ivar _context: A Context. @type _context: L{OpenSSL.SSL.Context} cXlg)z Initialize a L{_ContextFactoryWithContext} with a context. @param context: An SSL context. @type context: L{OpenSSL.SSL.Context} N_context)rXcontexts r8rY#_ContextFactoryWithContext.__init__s  r@cUR$)z Return the context created by L{_DeprecatedToCurrentPolicyForHTTPS._webContextFactory}. @return: A context. @rtype: L{OpenSSL.SSL.Context} rrms r8r%_ContextFactoryWithContext.getContexts}}r@rN)rErFrGrHrIrYrrJrDr@r8rrs r@rc$\rSrSrSrSrSrSrg)"_DeprecatedToCurrentPolicyForHTTPSia Adapt a web context factory to a normal context factory. @ivar _webContextFactory: An object providing a getContext method with C{hostname} and C{port} arguments. @type _webContextFactory: L{WebClientContextFactory} (or object with a similar C{getContext} method). cXlg)a Wrap a web context factory in an L{IPolicyForHTTPS}. @param webContextFactory: An object providing a getContext method with C{hostname} and C{port} arguments. @type webContextFactory: L{WebClientContextFactory} (or object with a similar C{getContext} method). N_webContextFactory)rXwebContextFactorys r8rY+_DeprecatedToCurrentPolicyForHTTPS.__init__s #4r@cNURRX5n[U5$)aK Called the wrapped web context factory's C{getContext} method with a hostname and port number and return the resulting context object. @param hostname: The hostname part of the URI. @type hostname: L{bytes} @param port: The port part of the URI. @type port: L{int} @return: A context factory. @rtype: L{IOpenSSLContextFactory} )rrr)rXrrSrs r8r3_DeprecatedToCurrentPolicyForHTTPS.creatorForNetlocs%))44XD)'22r@rNrrDr@r8rrs 43r@rcJ\rSrSrSr\S4SjrSrSrSr Sr S r S r S r g ) FileBodyProduceria L{FileBodyProducer} produces bytes from an input file object incrementally and writes them to a consumer. Since file-like objects cannot be read from in an event-driven manner, L{FileBodyProducer} uses a L{Cooperator} instance to schedule reads from the file. This process is also paused and resumed based on notifications from the L{IConsumer} provider being written to. The file is closed after it has been read, or if the producer is stopped early. @ivar _inputFile: Any file-like object, bytes read from which will be written to a consumer. @ivar _cooperate: A method like L{Cooperator.cooperate} which is used to schedule all reads. @ivar _readSize: The number of bytes to read from C{_inputFile} at a time. icjXlURUlX0lUR U5Ulgrt) _inputFile cooperate _cooperate _readSize_determineLengthlength)rX inputFile cooperatorreadSizes r8rYFileBodyProducer.__init__,s+#$..!++I6 r@cURnURnU"5nU"S[R 5 U"5nU"U[R 5 XT- $![a [s$f=f)z Determine how many bytes can be read out of C{fObj} (assuming it is not modified from this point on). If the determination cannot be made, return C{UNKNOWN_LENGTH}. r)seektellAttributeErrorr)osSEEK_ENDSEEK_SET)rXfObjrroriginalPositionends r8r!FileBodyProducer._determineLength2sg  "99D99D 6 Q f r{{+%%  "! ! "sAA-,A-cURR5 URR5 g![R a gf=f)zm Permanently stop writing bytes from the file to the consumer by stopping the underlying L{CooperativeTask}. N)rclose_taskstopr TaskFinishedrms r8 stopProducingFileBodyProducer.stopProducingCs?   JJOO      s7A Ac^TRTRU55TlTRR5nU4SjnUR SU5 U$)a* Start a cooperative task which will read bytes from the input file and write them to C{consumer}. Return a L{Deferred} which fires after all bytes have been written. If this L{Deferred} is cancelled before it is fired, stop reading and writing bytes. @param consumer: Any L{IConsumer} provider c>UR[R5(aTR5 O'UR[R 5(aOU$[R "5$rt)checkrCancelledErrorrr TaskStoppedr)reasonrXs r8 maybeStopped5FileBodyProducer.startProducing..maybeStoppedZsP||E0011""$d..// >># #r@cgrtrD)ignoreds r81FileBodyProducer.startProducing..estr@)r _writelooprwhenDone addCallbacks)rXconsumerdr s` r8startProducingFileBodyProducer.startProducingNsK__T__X%>? JJ   ! $ +\:r@c## URRUR5nU(dURR5 gUR U5 Sv M^7f)z Return an iterator which reads one chunk of bytes from the input file and writes them to the consumer for each time it is iterated. N)rreadrrwrite)rXrbytess r8rFileBodyProducer._writeloophsN OO((8E%%' NN5 !J sA A"c8URR5 g)z Temporarily suspend copying bytes from the input file to the consumer by pausing the L{CooperativeTask} which drives that activity. N)rpauserms r8pauseProducingFileBodyProducer.pauseProducingus r@c8URR5 g)z Undo the effects of a previous C{pauseProducing} and resume copying bytes to the consumer by resuming the L{CooperativeTask} which drives the write activity. N)rresumerms r8resumeProducing FileBodyProducer.resumeProducing|s r@)rrrrrN)rErFrGrHrIrrYrrrrrr"rJrDr@r8rrs1*.2E7 &" 4 r@rc2\rSrSrSrSrSSjrS SjrSrg) _HTTP11ClientFactoryiaE A factory for L{HTTP11ClientProtocol}, used by L{HTTPConnectionPool}. @ivar _quiescentCallback: The quiescent callback to be passed to protocol instances, used to return them to the connection pool. @ivar _metadata: Metadata about the low-level connection details, used to make the repr more useful. @since: 11.1 cXlX lgrt)_quiescentCallback _metadata)rXquiescentCallbackmetadatas r8rY_HTTP11ClientFactory.__init__s "3!r@cNSRURUR5$)Nz_HTTP11ClientFactory({}, {}))formatr'r(rms r8__repr___HTTP11ClientFactory.__repr__s$-44  # #T^^  r@c,[UR5$rt)rr')rXaddrs r8 buildProtocol"_HTTP11ClientFactory.buildProtocols#D$;$;<>%"6"677 .  #r@cp^^TRRT5nUU4SjnURU5 U$)z Do a request, and retry once (with a new connection) if it fails in a retryable manner. @param request: A L{Request} instance that will be requested using the wrapped protocol. c>TRTRURTR5(a#TR 5R U4Sj5$U$)Nc&>URT5$rt)request) connectionrLs r8rG_RetryingHTTP11ClientProtocol.request..failed..sz'9'9''Br@)rGrvaluerEr; addCallback)r rLrXs r8failed5_RetryingHTTP11ClientProtocol.request..failedsL  w?S?STT**,88B r@)r:rL addErrback)rXrLrrQs`` r8rL%_RetryingHTTP11ClientProtocol.requests3  ( ( 1  Vr@r9N) rErFrGrHrIrYrGrLrJrDr@r8r7r7s,8r@r7c~\rSrSrSr\rSrSrSr \ "5r SSjr SSjr SSjrS rS rS rS rg )HTTPConnectionPoolia A pool of persistent HTTP connections. Features: - Cached connections will eventually time out. - Limits on maximum number of persistent connections. Connections are stored using keys, which should be chosen such that any connections stored under a given key can be used interchangeably. Failed requests done using previously cached connections will be retried once if they use an idempotent method (e.g. GET), in case the HTTP server timed them out. @ivar persistent: Boolean indicating whether connections should be persistent. Connections are persistent by default. @ivar maxPersistentPerHost: The maximum number of cached persistent connections for a C{host:port} destination. @type maxPersistentPerHost: C{int} @ivar cachedConnectionTimeout: Number of seconds a cached persistent connection will stay open before disconnecting. @ivar retryAutomatically: C{boolean} indicating whether idempotent requests should be retried once if no response was received. @ivar _factory: The factory used to connect to the proxy. @ivar _connections: Map (scheme, host, port) to lists of L{HTTP11ClientProtocol} instances. @ivar _timeouts: Map L{HTTP11ClientProtocol} instances to a C{IDelayedCall} instance of their timeout. @since: 12.1 Tc8XlX l0Ul0Ulgrt)_reactor persistent _connections _timeouts)rXrr[s r8rYHTTPConnectionPool.__init__ s $r@c^^^TRRT5nU(aURS5nTRUR 5 TRU UR S:Xa:TR (aUUU4Sjn[XE5n[R"U5$U(aMTRTT5$)a Supply a connection, newly created or retrieved from the pool, to be used for one HTTP request. The connection will remain out of the pool (not available to be returned from future calls to this method) until one HTTP request has been completed over it. Afterwards, if the connection is still open, it will automatically be added to the pool. @param key: A unique key identifying connections that can be used interchangeably. @param endpoint: An endpoint that can be used to open a new connection if no cached connection is available. @return: A C{Deferred} that will fire with a L{HTTP11ClientProtocol} (or a wrapper) that can be used to send a single HTTP request. r QUIESCENTc(>TRTT5$rt)r;)endpointkeyrXsr8r2HTTPConnectionPool.getConnection..2sD,?,?X,Nr@) r\getrr]cancelstateretryAutomaticallyr7rsucceedr;)rXrcrb connectionsrMr=s``` r8 getConnection HTTPConnectionPool.getConnections0''++C0 $+J NN: & - - /z*;.**$NM!>""J}}Z00k""311r@cp^^UU4SjnTRU[U55nURU5nU$)z^ Create a new connection. This implements the new connection code path for L{getConnection}. c*>TRTU5 grt)_putConnection)rrcrXs r8r).quiescentCallbackCs   X .r@)_factoryreprconnect)rXrcrbr)rr?s`` r8r;!HTTPConnectionPool._newConnection:s9 /-- 14>B191A1A 2  r@cURR5 URURU5 URU g)z7 Remove a connection from the cache and disconnect it. N) transportloseConnectionr\remover])rXrcrMs r8_removeConnection$HTTPConnectionPool._removeConnectionLs; ++- #%%j1 NN: &r@cFURS:wa [S5eUR R U/5n[U5UR:XaUURS5nURR5 URUR5 URU URU5 URR!UR"UR$X5nXPRU'g![a URR S5 gf=f)z Return a persistent connection to the pool. This will be called by L{HTTP11ClientProtocol} when the connection becomes quiescent. r`z5BUG: Non-quiescent protocol added to connection pool.Nr)rg RuntimeError BaseException_logfailurer\ setdefaultrmaxPersistentPerHostrrvrwr]rfappendrZ callLatercachedConnectionTimeoutry)rXrcrMrjdroppedcids r8ro!HTTPConnectionPool._putConnectionTs   { * "K''223; { t88 8!ooa(G    , , . NN7 # * * ,w':&mm%%  ( ($*@*@# &)z"!  !!K   s C88%D D cb/nURR5H+nUH"nURUR55 M$ M- 0UlURR5HnUR 5 M 0Ul[ R"U5RS5$)z Close all persistent connections and remove them from the pool. @return: L{defer.Deferred} that fires when all connections have been closed. cgrtrD)igns r8r;HTTPConnectionPool.closeCachedConnections..sDr@) r\valuesrabortr]rfr gatherResultsrP)rXresults protocolsr7dcs r8closeCachedConnections)HTTPConnectionPool.closeCachedConnectionsps**113Iqwwy)4..'')B IIK*""7+778HIIr@)r\rZr]r[N)T)rcrrbrr4zDeferred[HTTP11ClientProtocol])rErFrGrHrIr%rqrrrhrr~rYrkr;ryrorrJrDr@r8rVrVsx$L$H! 8D &2&2'<&2 '&2P'< '$')8Jr@rVcR\rSrSrSrSSjrSrS SjrSrg) _AgentBaseia Base class offering common facilities for L{Agent}-type classes. @ivar _reactor: The C{IReactorTime} implementation which will be used by the pool, and perhaps by subclasses as well. @ivar _pool: The L{HTTPConnectionPool} used to manage HTTP connections. c:Uc [US5nXlX lgNF)rVrZ_pool)rXrpools r8rY_AgentBase.__init__s <%gu5D  r@cb[[U55(aSU-S-nX4S;aU$SX#4-$)zw Compute the string to use for the value of the I{Host} header, based on the given scheme, host name, and port number. [])httpr`)r^r_s%b:%d)rr)rXrOrRrSs r8_computeHostValue_AgentBase._computeHostValues? d+ , ,$;%D >= =K4,&&r@c^^^^^^[T[5(d[ST<S[T5S35e[ T5mTc [ 5mTR S5(dQTR5mTRSTRTRTRTR55 TRRX5nSUUUUUU4Sjjn URU 5$)zS Issue a new request, given the endpoint and the path sent as part of the request. zmethod=z is z, but must be bytessHostc |>UR[R"TTTTTRRTS95$)N)r[ parsedURI)rLr1 _constructrr[)protorEheadersrr requestPathrXs r8 cbConnected4_AgentBase._requestWithEndpoint..cbConnecteds>=="" #zz44'   r@)rrr4Deferred[IResponse])rBr TypeErrortyper%r' hasHeadercopy addRawHeaderrrOrRrSrrkrP) rXrcrbrrrrErrrs ` ````` r8_requestWithEndpoint_AgentBase._requestWithEndpoints&%((gfZtDL>ATUV V#F+ ?iG  ))llnG  &&$$inninn  JJ $ $S 3  }}[))r@)rrZN)rrrzHTTPConnectionPool | Noner4None)rcztuple[bytes, bytes, int]rbrrrrrLrzHeaders | NonerEzIBodyProducer | Nonerrr4r) rErFrGrHrIrYrrrJrDr@r8rrsd '-* %-*(-* -*  -*  -*+-*-* -*r@rc(\rSrSrSrSrSSjrSrg)_StandardEndpointFactoryia Standard HTTP endpoint destinations - TCP for HTTP, TCP+TLS for HTTPS. @ivar _policyForHTTPS: A web context factory which will be used to create SSL context objects for any SSL connections the agent needs to make. @ivar _connectTimeout: If not L{None}, the timeout passed to L{HostnameEndpoint} for specifying the connection timeout. @ivar _bindAddress: If not L{None}, the address passed to L{HostnameEndpoint} for specifying the local address to bind to. c4XlX lX0lX@lg)a' @param reactor: A provider to use to create endpoints. @type reactor: see L{HostnameEndpoint.__init__} for acceptable reactor types. @param contextFactory: A factory for TLS contexts, to control the verification parameters of OpenSSL. @type contextFactory: L{IPolicyForHTTPS}. @param connectTimeout: The amount of time that this L{Agent} will wait for the peer to accept a connection. @type connectTimeout: L{float} or L{None} @param bindAddress: The local address for client sockets to bind to. @type bindAddress: L{bytes} or L{None} N)rZr_connectTimeout _bindAddress)rXrrconnectTimeout bindAddresss r8rY!_StandardEndpointFactory.__init__s" --'r@c0nURbURUS'URUS'[UR5n[URX1R40UD6nURS;aU$URS;a;URRURUR5n[XT5$[SUR<35e![a [ SR US95ef=f)z Connect directly over TCP for C{b'http'} scheme, and TLS for C{b'https'}. @param uri: L{URI} to connect to. @return: Endpoint to connect to. timeoutrzdThe host of the provided URI ({uri.host!r}) contains non-ASCII octets, it should be ASCII decodable.rh>wsrr\zUnsupported scheme: )rrrrRUnicodeDecodeErrorrfr-rrZrSrOrrrr)rXrhrrRrbconnectionCreators r8endpointForURI'_StandardEndpointFactory.endpointForURIs    + $ 4 4F9  $ 1 1} )D$DMM4LVL ::) )O ZZ- - $ 4 4 E E#((! !!2= =$';CJJ>%JK K%" !&S&/   s C#C>)rrrrZN)rhrLr4r)rErFrGrHrIrYrrJrDr@r8rrs (,"Lr@rcx\rSrSrSr\"5SSS4Sjr\S Sj5rSr Sr S S Sjjr S r g) Agentia L{Agent} is a very basic HTTP client. It supports I{HTTP} and I{HTTPS} scheme URIs. @ivar _pool: An L{HTTPConnectionPool} instance. @ivar _endpointFactory: The L{IAgentEndpointFactory} which will be used to create endpoints for outgoing connections. @since: 9.0 Nc[R"U5(d1[R"[ U5S-S[ S9 [ U5n[XX45nURXU5 g)au Create an L{Agent}. @param reactor: A reactor for this L{Agent} to place outgoing connections. @type reactor: see L{HostnameEndpoint.__init__} for acceptable reactor types. @param contextFactory: A factory for TLS contexts, to control the verification parameters of OpenSSL. The default is to use a L{BrowserLikePolicyForHTTPS}, so unless you have special requirements you can leave this as-is. @type contextFactory: L{IPolicyForHTTPS}. @param connectTimeout: The amount of time that this L{Agent} will wait for the peer to accept a connection. @type connectTimeout: L{float} @param bindAddress: The local address for client sockets to bind to. @type bindAddress: L{bytes} @param pool: An L{HTTPConnectionPool} instance, or L{None}, in which case a non-persistent L{HTTPConnectionPool} instance will be created. @type pool: L{HTTPConnectionPool} z was passed as the HTTPS policy for an Agent, but it does not provide IPolicyForHTTPS. Since Twisted 14.0, you must pass a provider of IPolicyForHTTPS.rW) stacklevelcategoryN) r- providedBywarningswarnrrDeprecationWarningrr_init)rXrrrrrendpointFactorys r8rYAgent.__init__%shD)).99 MM^$66+  @ON2 ^  7T2r@cLURU5nURXU5 U$)aw Create a new L{Agent} that will use the endpoint factory to figure out how to connect to the server. @param reactor: A reactor for this L{Agent} to place outgoing connections. @type reactor: see L{HostnameEndpoint.__init__} for acceptable reactor types. @param endpointFactory: Used to construct endpoints which the HTTP client will connect with. @type endpointFactory: an L{IAgentEndpointFactory} provider. @param pool: An L{HTTPConnectionPool} instance, or L{None}, in which case a non-persistent L{HTTPConnectionPool} instance will be created. @type pool: L{HTTPConnectionPool} @return: A new L{Agent}. )__new__r)rgrrragents r8usingEndpointFactoryAgent.usingEndpointFactoryVs%, C  Gd3 r@c<[RXU5 X lg)a- Initialize a new L{Agent}. @param reactor: A reactor for this L{Agent} to place outgoing connections. @type reactor: see L{HostnameEndpoint.__init__} for acceptable reactor types. @param endpointFactory: Used to construct endpoints which the HTTP client will connect with. @type endpointFactory: an L{IAgentEndpointFactory} provider. @param pool: An L{HTTPConnectionPool} instance, or L{None}, in which case a non-persistent L{HTTPConnectionPool} instance will be created. @type pool: L{HTTPConnectionPool} @return: A new L{Agent}. N)rrY_endpointFactory)rXrrrs r8r Agent._initps( D40 /r@c8URRU5$)z Get an endpoint for the given URI, using C{self._endpointFactory}. @param uri: The URI of the request. @type uri: L{URI} @return: An endpoint which can be used to connect to given address. )rr)rXrhs r8 _getEndpointAgent._getEndpoints$$33C88r@c f[UR55n[RU5nUR U5nURURUR4nURUUUUUUUR5$![ a! [ R"[55s$f=f)z Issue a request to the server indicated by the given C{uri}. An existing connection from the connection pool may be used or a new one may be created. I{HTTP} and I{HTTPS} schemes are supported in C{uri}. @see: L{twisted.web.iweb.IAgent.request} )r&rQrLrjrrrfailr"rOrRrSrrr)rXrrhrrErrbrcs r8rL Agent.requests"ciik*MM#&  )((3H@((          " )::gi( ( )sB(B0/B0)rrtNN rrrhrrzOptional[Headers]rEzOptional[IBodyProducer]r4r) rErFrGrHrIrrYrurrrrLrJrDr@r8rrs~ 12  /3b20. 9&*04      #  .     r@rc,\rSrSrSrSSjrSSjrSrg) ProxyAgentiz{ An HTTP agent able to cross HTTP proxies. @ivar _proxyEndpoint: The endpoint used to connect to the proxy. @since: 11.1 NcNUcSSKJn [RXU5 Xlg)Nrr)rrrrY_proxyEndpoint)rXrbrrs r8rYProxyAgent.__init__s ? 0D40&r@c [UR55nSUR4nURUURU[R U5UUU5$)z/ Issue a new request via the configured proxy. z http-proxy)r&rQrrrLrj)rXrrhrrErcs r8rLProxyAgent.requests_ciik*T001 ((      MM#      r@)rr)rErFrGrHrIrYrLrJrDr@r8rrs'  r@rcz\rSrSr%SrS\S'S\S'S\S'S\S'SS jrS rS rS r SSjr Sr Sr Sr Srg )_FakeStdlibRequestiaQ A fake L{urllib.request.Request} object for L{cookiejar} to work with. @see: U{urllib.request.Request } @ivar uri: Request URI. @ivar headers: Request headers. @ivar type: The scheme of the URI. @ivar host: The host[:port] of the URI. @since: 11.1 r5rhrrRr'_twistedHeadersc[U5Ul[5Ul[R U5n[UR 5Ul[UR5UlUR UR4S;a+U=RS[UR5-- sl[UR5Ul SUl g)z3 Create a fake request. @param uri: Request URI. r:FN) rrhr'rrLrjrOrrRrSr5origin_req_host unverifiable)rXrh_uris r8rY_FakeStdlibRequest.__init__s  $&y}}S! -  + KK #+K K IIs499~- -I+DII6!r@cJURR[U55$rt)rrr)rXheaders r8 has_header_FakeStdlibRequest.has_header s##--mF.CDDr@c`URR[U5[U55 grt)rrr)rXnamerOs r8add_unredirected_header*_FakeStdlibRequest.add_unredirected_header s" ))-*=}U?STr@cUR$rtrrms r8 get_full_url_FakeStdlibRequest.get_full_urls xxr@NcURR[U5U5nUb UVs/sHn[U5PM nnUS$gs snf)Nr)r getRawHeadersrr)rXrdefaultrxs r8 get_header_FakeStdlibRequest.get_headersO&&44]45H'R  0781|AG81: 9sA cUR$rt)rRrms r8get_host_FakeStdlibRequest.get_host yyr@cUR$rt)rrms r8get_type_FakeStdlibRequest.get_typerr@cgrrDrms r8is_unverifiable"_FakeStdlibRequest.is_unverifiablesr@)rrRrrrrh)rhrr4rrt)rErFrGrHrI__annotations__rYrrrrrrr rJrDr@r8rrsH" H I I"(EUr@rc*\rSrSr%S\S'SSjrSrg)_FakeUrllibResponseInfoi$r.responsecURRR[U5U5nUVs/sHn[ U5PM nnU$s snfrt)rrrrr)rXrrrrhs r8get_all_FakeUrllibResponseInfo.get_all(sE--''55mD6I7S&- .g\!_g . /sA rDN)rr5rrr4z list[str])rErFrGrHr rrJrDr@r8r r $s r@r c8\rSrSr%SrS\S'S SjrS SjrSrg) _FakeStdlibResponsei.z A fake L{urllib.response.Response} object for L{http.cookiejar} to work with. @ivar response: Underlying Twisted Web response. @since: 11.1 r.rcXlgrtrrXrs r8rY_FakeStdlibResponse.__init__:s r@c0[UR5nU$rt)r r)rXr?s r8info_FakeStdlibResponse.info=s(7 r@rN)rr.r4r)r4 _InfoType) rErFrGrHrIr rYrrJrDr@r8rr.s!r@rcr\rSrSr%SrS\S'S\S'S SjrS SS jjrSS jrS r g) CookieAgentiBa L{CookieAgent} extends the basic L{Agent} to add RFC-compliant handling of HTTP cookies. Cookies are written to and extracted from a L{CookieJar} instance. The same cookie jar instance will be used for any requests through this agent, mutating it whenever a I{Set-Cookie} header appears in a response. @ivar _agent: Underlying Twisted Web agent to issue requests through. @ivar cookieJar: Initialized cookie jar to read cookies from and store cookies to. @since: 11.1 r*_agentr cookieJarcXlX lgrtrr )rXrr s r8rYCookieAgent.__init__Ws  "r@NcUbUO [5n[U5nURS5(d[URR U5 UR SS5nUb+UR 5nURS[U55 URRXXT5RURU5$)aU Issue a new request to the wrapped L{Agent}. Send a I{Cookie} header if a cookie for C{uri} is stored in L{CookieAgent.cookieJar}. Cookies are automatically extracted and stored from requests. If a C{'cookie'} header appears in C{headers} it will override the automatic cookie header obtained from the cookie jar. @see: L{Agent.request} NCookieCookie) r'rrr add_cookie_headerrrrrrrLrP_extractCookies)rXrrhrrE actualHeaders lastRequest cookieHeaders r8rLCookieAgent.request[s&$+#6GI (- &&y11 NN , ,[ 9&11(DAL' - 2 2 4 **9mL6QR{{""  +d**K 8 9r@cPURR[U5U5 U$)z Extract response cookies and store them in the cookie jar. @param response: the Twisted Web response that we are processing. @param request: A L{_FakeStdlibRequest} wrapping our Twisted request, for L{CookieJar} to extract cookies from. )r extract_cookiesr)rXrrLs r8r(CookieAgent._extractCookies}s# &&':8'DgNr@r")rr*r rr4rrr)rr.rLrr4r.) rErFrGrHrIr rYrLr(rJrDr@r8rrBst N#&*04 9 9 9# 9 . 9  9D ! ,>   r@rc$\rSrSrSrSrSrSrg) GzipDecoderiz A wrapper for a L{Response} instance which handles gzip'ed body. @ivar original: The original L{Response} object. @since: 11.1 c&Xl[Ulgrt)originalr)rrs r8rYGzipDecoder.__init__s $ r@c`URR[XR55 g)zN Override C{deliverBody} to wrap the given C{protocol} with L{_GzipProtocol}. N)r3 deliverBody _GzipProtocol)rXrs r8r6GzipDecoder.deliverBodys !!---"HIr@)rr3N)rErFrGrHrIrYr6rJrDr@r8r1r1s%Jr@r1c*\rSrSrSrSrSrSrSrg)r7ia A L{Protocol} implementation which wraps another one, transparently decompressing received data. @ivar _zlibDecompress: A zlib decompress object used to decompress the data stream. @ivar _response: A reference to the original response, in case of errors. @since: 11.1 ctXlX l[R"S[R-5Ulg)N)r3 _responsezlib decompressobj MAX_WBITS_zlibDecompress)rXrrs r8rY_GzipProtocol.__init__s( !#11"t~~2EFr@cURRU5nU(aURRU5 gg![Ra [ [ 5/UR 5ef=f)zb Decompress C{data} with the zlib decompressor, forwarding the raw data to the original protocol. N) r@ decompressr=r#rr"r<r3 dataReceived)rXdatarawDatas r8rD_GzipProtocol.dataReceivedsc  >**55d;G  MM & &w / zz > ')dnn= = >s A4A5c$URR5nU(aURRU5 URRU5 g![Ra! [ U[ 5/UR 5ef=f)zZ Forward the connection lost event, flushing remaining data from the decompressor if any. N) r@flushr=r#rr"r<r3rDconnectionLost)rXr rFs r8rJ_GzipProtocol.connectionLostst  F**002G  MM & &w / $$V, zz F &')!4dnnE E Fs A5B)r<r@r3N rErFrGrHrIrYrDrJrJrDr@r8r7r7s G 0 -r@r7c.\rSrSrSrSrSSjrSrSrg) ContentDecoderAgentiaE An L{Agent} wrapper to handle encoded content. It takes care of declaring the support for content in the I{Accept-Encoding} header and automatically decompresses the received data if the I{Content-Encoding} header indicates a supported encoding. For example:: agent = ContentDecoderAgent(Agent(reactor), [(b'gzip', GzipDecoder)]) @param agent: The agent to wrap @type agent: L{IAgent} @param decoders: A sequence of (name, decoder) objects. The name declares which encoding the decoder supports. The decoder must accept an L{IResponse} and return an L{IResponse} when called. The order determines how the decoders are advertised to the server. Names must be unique.not be duplicated. @type decoders: sequence of (L{bytes}, L{callable}) tuples @since: 11.1 @see: L{GzipDecoder} cXl[U5UlSRUVs/sHo3SPM sn5Ulgs snf)N,r)rdict _decodersjoin _supported)rXrdecodersdecoders r8rYContentDecoderAgent.__init__s6 h))x$HxGQZx$HI$HsANcUc [5nOUR5nURSUR5 URR XX45nUR UR5$)z^ Send a client request which declares supporting compressed content. @see: L{Agent.request}. sAccept-Encoding)r'rrrTrrLrP_handleResponserXrrhrrEdeferreds r8rLContentDecoderAgent.requests] ?iGllnG/A;;&&vGJ##D$8$899r@cURRS/5nSRU5RS5nU(a`UR 5R 5nUR RU5nUb U"U5nOURU5 O U(aM`U(a.URRSSRU5/5 U$URRS5 U$)zH Check if the response is encoded, and wrap it to handle decompression. sContent-EncodingrP) rrrSsplitrrQrRrer setRawHeaders removeHeader)rXrcontentEncodingHeadersrrVs r8rY#ContentDecoderAgent._handleResponses"*!1!1!?!?@SUW!X!%+A!B!H!H!N$)--/557Dnn((.G""8,'--d3%$ "    * *#dii0F&G%H      ) )*= >r@)rrRrTr) rErFrGrHrIrYrLrYrJrDr@r8rNrNs6J :r@rN)s Authorizationr%sCookie2sProxy-AuthorizationsWWW-Authenticatec\rSrSrSr\R \R\R\R/r \R/r S S Sjjr S SjrSrSrSrS rg) RedirectAgenti!a An L{Agent} wrapper which handles HTTP redirects. The implementation is rather strict: 301 and 302 behaves like 307, not redirecting automatically on methods different from I{GET} and I{HEAD}. See L{BrowserLikeRedirectAgent} for a redirecting Agent that behaves more like a web browser. @param redirectLimit: The maximum number of times the agent is allowed to follow redirects before failing with a L{error.InfiniteRedirection}. @param sensitiveHeaderNames: An iterable of C{bytes} enumerating the names of headers that must not be transmitted when redirecting to a different origins. These will be consulted in addition to the protocol-specified set of headers that contain sensitive information. @cvar _redirectResponses: A L{list} of HTTP status codes to be redirected for I{GET} and I{HEAD} methods. @cvar _seeOtherResponses: A L{list} of HTTP status codes to be redirected for any method and the method altered to I{GET}. @since: 11.1 cXlX lUVs1sHn[U5iM nnUR[5 XPlgs snfrt)r_redirectLimit_canonicalHeaderNameupdate_defaultSensitiveHeaders_sensitiveHeaderNames)rXr redirectLimitsensitiveHeaderNameseach sensitives r8rYRedirectAgent.__init__EsF  +URT5 U$rt)setPreviousResponse) newResponsers r8_chainResponse5RedirectAgent._handleRedirect.._chainResponses  + +H 5 r@rb)rfr#InfiniteRedirectioncoderr"rrRedirectWithNoLocationrurLrjrOrRrSr'getAllRawHeadersrjrrLrPrY)rXrrrhr redirectCounterrlocationHeadersrtrparsedLocation sameOriginrawNamerawValuer[r|s ` r8_handleRedirectRedirectAgent._handleRedirectis // /++ ?#C!'#,: :"**88bI.. :CC!'#,: :((a.@A  c*I ]]84N!!^%:%::<^^~':'::<^^~':'::  !291I1I1K1K-G"*D*DD*)1K;;&&vA  ^,##  &w8I  s G 8G c2URUR;aNUS;a5[R"URUS9n[ [ U5/U5eUR XX4U5$URUR;aUR USX4U5$U$)zI Handle the response, making another request if it indicates a redirect. )r@rArxr@)r_redirectResponsesr# PageRedirectrr"r_seeOtherResponses)rXrrrhrrrs r8rYRedirectAgent._handleResponses ==D33 3..((E$gcl^X>>''# V V ]]d55 5''&# V Vr@)rrfrj)rrD)rr*rkrerlzIterable[bytes]r)rErFrGrHrIr$MOVED_PERMANENTLYFOUNDTEMPORARY_REDIRECTPERMANENT_REDIRECTr SEE_OTHERrrYrLrurrYrJrDr@r8rdrd!s6      ..)  02 / / /. /S .* X r@rdc\rSrSrSr\R /r\R\R\R\R/r Sr g)BrowserLikeRedirectAgentia/ An L{Agent} wrapper which handles HTTP redirects in the same fashion as web browsers. Unlike L{RedirectAgent}, the implementation is more relaxed: 301 and 302 behave like 303, redirecting automatically on any method and altering the redirect request to a I{GET}. @see: L{RedirectAgent} @since: 13.1 rDN)rErFrGrHrIr$rrrrrrrrJrDr@r8rrs@ 112     r@rc*\rSrSrSrSrSrSrSrg)_ReadBodyProtocoliaG Protocol that collects data sent to it. This is a helper for L{IResponse.deliverBody}, which collects the body and fires a deferred with it. @ivar deferred: See L{__init__}. @ivar status: See L{__init__}. @ivar message: See L{__init__}. @ivar dataBuffer: list of byte-strings received @type dataBuffer: L{list} of L{bytes} c6X0lXlX l/Ulg)z @param status: Status of L{IResponse} @ivar status: L{int} @param message: Message of L{IResponse} @type message: L{bytes} @param deferred: deferred to fire when response is complete @type deferred: L{Deferred} firing with L{bytes} N)r[statusmessage dataBuffer)rXrrr[s r8rY_ReadBodyProtocol.__init__s!   r@c:URRU5 g)z/ Accumulate some more bytes from the response. N)rr)rXrEs r8rD_ReadBodyProtocol.dataReceiveds t$r@c UR[5(a5URRSR UR 55 gUR[ 5(aTURR[URURSR UR 555 gURRU5 g)z Deliver the accumulated response bytes to the waiting L{Deferred}, if the response body has been completely received without error. r@N) rrr[callbackrSrrerrbackrBrr)rXr s r8rJ _ReadBodyProtocol.connectionLosts << % % MM " "388DOO#< = \\+ , , MM ! !$KKsxx/H  MM ! !& )r@)rr[rrNrLrDr@r8rrs  % *r@rc^^SU4Sjjn[R"U5n[URURU5mU4SjmUR T5 TR b"T"5c[R"S[SS9 U$)a Get the body of an L{IResponse} and return it as a byte string. This is a helper function for clients that don't want to incrementally receive the body of an HTTP response. @param response: The HTTP response for which the body will be read. @type response: L{IResponse} provider @return: A L{Deferred} which will fire with the body of the response. Cancelling it will close the connection to the server immediately. c*>T"5nUbU"5 gg)z Cancel a L{readBody} call, close the connection to the HTTP server immediately, if it is still open. @param deferred: The cancelled L{defer.Deferred}. NrD)r[rgetAborts r8rfreadBody..cancels    G r@c2>[TRSS5$)NabortConnection)getattrrv)rsr8rreadBody..getAbort sx))+rs #  !$>>FF&223+F"=7+#F:796LMI$ 5;;ood68D    1 >#H#HL _XXX: Ir1a  2q!$- eDk!  &&$$  _===@ #$%: _$3$3$3N ]lll^=8++=4==@`J`JFH*H*V "#FLFL$FLR VZ JZ Z z V$ $ $ NDDN i -( VFFFRJ#I.J*)-%i0)-X VDDDN$**$ V~~~B}.3*))3*l) X O0 Cs.L::MM