hR.pSrSSKJr SSKrSSKJr SSKJr SSKJrJ r SSK J r SSK J r SS KJr SS KJr \R$"S \R&5r\R$"S \R*5r\R$"S \R.5r\R$"S\R*\R&-5r\R$"S\R*\R&-5r\R$"S\R*5rSrSSSjjrS S!SjjrS"S#Sjjr\R$"S\R*5r S S$Sjjr!S%S&Sjjr"S'S(Sjjr#S)S*Sjjr$S+S,Sjjr%S-S.Sjjr&S/S0Sjjr'S1Sjr(g)2z( Functions for dealing with markup text ) annotationsN)Iterable)name2codepoint)MatchPattern)urljoin) StrOrBytes)safe_url_string) to_unicodezI&((?P[a-z\d]+)|#(?P\d+)|#x(?P[a-f\d]+))(?P;?)z<[a-zA-Z\/!].*?>z5]*href\s*=\s*[\"\']\s*([^\"\'\s]+)\s*[\"\']z}]*http-equiv[^>]*refresh[^>]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)z]*content\s*=\s*(?P["\'])(?P(\d*\.)?\d+)\s*;\s*url=\s*(?P.*?)(?P=quote)[^>]*?\shttp-equiv\s*=[^>]*refreshz<((?P.*?)(?P\]\]>))z cV^^SUU4Sjjn[RU[X55$)ucRemove entities from the given `text` by converting them to their corresponding unicode character. `text` can be a unicode string or a byte string encoded in the given `encoding` (which defaults to 'utf-8'). If `keep` is passed (with a list of entity names) those entities will be kept (they won't be removed). It supports both numeric entities (``&#nnnn;`` and ``&#hhhh;``) and named entities (such as `` `` or ``>``). If `remove_illegal` is ``True``, entities that can't be converted are removed. If `remove_illegal` is ``False``, entities that can't be converted are kept "as is". For more information see the tests. Always returns a unicode string (with the entities removed). >>> import w3lib.html >>> w3lib.html.replace_entities(b'Price: £100') 'Price: \xa3100' >>> print(w3lib.html.replace_entities(b'Price: £100')) Price: £100 >>> c>UR5nSnURS5(a[USS5nOURS5(a[USS5nOURS5(akUSnUR5T;aUR S5$[ R"U5=(d$ [ R"UR55nUb7SUs=::aS::aO O[ U45RS 5$[U5$T(aURS 5(aS $UR S5$![[4a NBf=f) Ndec hexnamedrcp1252 semicolon) groupdictgetintlowergrouprbytesdecodechr ValueError OverflowError)mgroupsnumber entity_namekeepremove_illegals Q/root/1688_scrapy/alibaba-scraper/venv/lib/python3.13/site-packages/w3lib/html.pyconvert_entity(replace_entities..convert_entityHs0 ::e  +F ZZ  +F ZZ  /K  "d*wwqz!#'' 48J8J!!#9F    6)T) &+228<<6{"$ ;(?(?rOQWWQZO .  s%*E  E EEr"z Match[str]returnstr)_ent_resubr )textr&r'encodingr)s `` r(replace_entitiesr2's(BPP8 ;;~z$'A BBcP[[R[X555$N)boolr.searchr )r0r1s r( has_entitiesr8gs z$9: ;;r3c@[RU[X55$)a3Replace all markup tags found in the given `text` by the given token. By default `token` is an empty string so it just removes all tags. `text` can be a unicode string or a regular string encoded as `encoding` (or ``'utf-8'`` if `encoding` is not given.) Always returns a unicode string. Examples: >>> import w3lib.html >>> w3lib.html.replace_tags('This text contains some tag') 'This text contains some tag' >>> w3lib.html.replace_tags('

Je ne parle pas fran\xe7ais

', ' -- ', 'latin-1') ' -- Je ne parle pas -- fran\xe7ais -- -- ' >>> )_tag_rer/r )r0tokenr1s r( replace_tagsr<ks( ;;uj8 99r3z|$)cD[X5n[RSU5$)zRemove HTML Comments. >>> import w3lib.html >>> w3lib.html.remove_comments(b"test whatever") 'test whatever' >>> r)r _REMOVECOMMENTS_REr/)r0r1utexts r(remove_commentsr@s! t &E  ! !"e ,,r3c^^^T(aT(a [S5eTVs1sHoDR5iM snmTVs1sHoDR5iM snmSUU4SjjmSU4SjjnSn[R"U[R[R -5nUR U[X55$s snfs snf)aRemove HTML Tags only. `which_ones` and `keep` are both tuples, there are four cases: ============== ============= ========================================== ``which_ones`` ``keep`` what it does ============== ============= ========================================== **not empty** empty remove all tags in ``which_ones`` empty **not empty** remove all tags except the ones in ``keep`` empty empty remove all tags **not empty** **not empty** not allowed ============== ============= ========================================== Remove all tags: >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags(doc) 'This is a link: example' >>> Keep only some tags: >>> w3lib.html.remove_tags(doc, keep=('div',)) '
This is a link: example
' >>> Remove only specific tags: >>> w3lib.html.remove_tags(doc, which_ones=('a','b')) '

This is a link: example

' >>> You can't remove some and keep some: >>> w3lib.html.remove_tags(doc, which_ones=('a',), keep=('p',)) Traceback (most recent call last): ... ValueError: Cannot use both which_ones and keep >>> z#Cannot use both which_ones and keepcF>UR5nT(aUT;$UT;$r5)r)tagr& which_oness r( will_remove remove_tags..will_removes&iik *$ $$r3cf>URS5nT"U5(aS$URS5$)Nrr)r)r"rCrEs r( remove_tagremove_tags..remove_tags-ggaj %%r51771:5r3z/]+).*?>)rCr-r,r6r+)r rrecompileDOTALL IGNORECASEr/r ) r0rDr&r1rCrIregexretagsrEs `` @r( remove_tagsrQsbd>??)34#))+4J#' (4CIIK4 (D 6 E ZZryy2==8 9F ::j*T"< ==!5 (s CCc [X5nU(atSRUVs/sHnSUSUSUS3PM sn5n[R"U[R[R -5nUR SU5nU$s snf)aRemove tags and their content. `which_ones` is a tuple of which tags to remove including their content. If is empty, returns the string unmodified. >>> import w3lib.html >>> doc = '

This is a link: example

' >>> w3lib.html.remove_tags_with_content(doc, which_ones=('b',)) '' >>> ||r)r joinrKrLrMrNr/)r0rDr1r?rCtagsrPs r(remove_tags_with_contentrWs{ t &Exx S AcU'#c#e< STD"))bmm";< 2u% LTsB cd[X5nUHnURU[X#55nM U$)aRemove escape characters. `which_ones` is a tuple of which escape characters we want to remove. By default removes ``\n``, ``\t``, ``\r``. `replace_by` is the string to replace the escape characters by. It defaults to ``''``, meaning the escape characters are removed. )r replace)r0rD replace_byr1r?ecs r(replace_escape_charsr\s2 t &E b*Z"BC Lr3cSSjn[X5nSnU"U[5H:n[U[5(aU[ XqUS9- nM'XgR S5- nM< U$)aD This function receives markup as a text (always a unicode string or a UTF-8 encoded string) and does the following: 1. removes entities (except the ones in `keep`) from any part of it that is not inside a CDATA 2. searches for CDATAs and extracts their text (if any) without modifying it. 3. removes the found CDATAs c3# SnURU5H"nURS5upEXUv Uv UnM$ XSv g7f)NrrH)finditerspan)txtpatternoffsetmatchmatch_smatch_es r(_get_fragments&unquote_markup.._get_fragmentssP%%c*E$zz!} GW% %KF + 'lsAAr)r&r'cdata_d)rar-rbz Pattern[str]r,zIterable[str | Match[str]])r _cdata_re isinstancer-r2r)r0r&r'r1rgr?ret_textfragments r(unquote_markuprnsg" t &EH"5)4 h $ $ (N H y1 1H5 Or3c[XS9n[RU5=n(a,[[ U5[ UR S5US95$[ U5$)zReturn the base url if declared in the given HTML `text`, relative to the given base url. If no base url is found, the given `baseurl` is returned. )r1rH)r@ _baseurl_rer7rr r)r0baseurlr1r?r"s r( get_base_urlrr0sW!9E   u %%q% G $oaggaj8&T   7 ##r3c[X5n[XC5n[ [ U55n[ RU5=(d [RU5=n(aR[URS55n[URS5RS5U5n[X5nXg4$g![a [U5 ef=f)aDReturn the http-equiv parameter of the HTML meta element from the given HTML text and return a tuple ``(interval, url)`` where interval is an integer containing the delay in seconds (or zero if not present) and url is a string with the absolute url to redirect. If no meta redirect is found, ``(None, None)`` is returned. rurlz "')NN)r UnicodeDecodeErrorprintrWr@r2_meta_refresh_rer7_meta_refresh_re2floatrr stripr)r0rqr1 ignore_tagsr?r"intervalrts r(get_meta_refreshr}Bs4* %U 8E ,U3 4E  # #E * M.?.F.Fu.MMqM(aggen226:HEg#}   d  s B77Cc,UR[5$)a Strip all leading and trailing space characters (as defined in https://www.w3.org/TR/html5/infrastructure.html#space-character). Such stripping is useful e.g. for processing HTML element attributes which contain URLs, like ``href``, ``src`` or form ``action`` - HTML5 standard defines them as "valid URL potentially surrounded by spaces" or "valid non-empty URL potentially surrounded by spaces". >>> strip_html5_whitespace(' hello\n') 'hello' )rzHTML5_WHITESPACE)r0s r(strip_html5_whitespacer`s ::& ''r3)Tutf-8) r0r r& Iterable[str]r'r6r1r-r,r-r5)r0r r1 str | Noner,r6)rN)r0r r;r-r1rr,r-)r0r r1rr,r-)rrN) r0r rDrr&rr1rr,r-)rN)r0r rDrr1rr,r-))   rN) r0r rDrrZr r1rr,r-)rTN) r0r r&rr'r6r1rr,r-)rr)r0r rqr r1r-r,r-)rr)scriptnoscript) r0r rqr-r1r-r{rr,z%tuple[None, None] | tuple[float, str])r0r-r,r-))__doc__ __future__rrKcollections.abcr html.entitiesrrr urllib.parser w3lib._typesr w3lib.urlr w3lib.utilr rLrNr.rMr:Irprwrxrjrr2r8r<r>r@rQrWr\rnrrr}rrr3r(rs# $( #%! **PMM  **(")) 4jjQSUSWSWX ::EII JJLII  JJCRYY  !  =C =C =C=C =C  =C@<:.ZZ 2BII> - !# D> D>D> D> D>  D>PNR "/@J4!3     0 % % %% %  %RAH$ $)$:=$$(!7     + < (r3