+h2^SSKJr SSKrSSKrSSKJrJr SSKJr SSK J r SSK J r J r JrJrJrJrJr SSKJrJr SSKJr \ (aSS KJrJrJr SS KJr \R<"\5r \"S S S 9r!"SS \\5r""SS\"5r#"SS\$\ 5r%\"SS9"SS55r&SSjr'g)) annotationsN)ABCabstractmethod) dataclass)Enum) TYPE_CHECKINGAnyCallableLiteralOptionalTypeVarUnion)BaseDocumentTransformerDocument)Self) CollectionIterableSequence)SetTS TextSplitter)boundc\rSrSrSrSS\SSS4SSjjr\SSj5rSSS jjr SS jr SS jr SS jr \ SSj5r\ SS \"5S4SSjj5rSSjrSrg )rz)Interface for splitting text into chunks.iFTcUS::aSU3n[U5eUS:aSU3n[U5eX!:aSUSUS3n[U5eXlX lX0lX@lXPlX`lg)aCreate a new TextSplitter. Args: chunk_size: Maximum size of chunks to return chunk_overlap: Overlap in characters between chunks length_function: Function that measures the length of given chunks keep_separator: Whether to keep the separator and where to place it in each corresponding chunk (True='start') add_start_index: If `True`, includes chunk's start index in metadata strip_whitespace: If `True`, strips whitespace from the start and end of every document rzchunk_size must be > 0, got z chunk_overlap must be >= 0, got zGot a larger chunk overlap (z) than chunk size (z), should be smaller.N) ValueError _chunk_size_chunk_overlap_length_function_keep_separator_add_start_index_strip_whitespace)self chunk_size chunk_overlaplength_functionkeep_separatoradd_start_indexstrip_whitespacemsgs M/root/34ku/venv/lib/python3.13/site-packages/langchain_text_splitters/base.py__init__TextSplitter.__init__!s* ?0 =CS/ ! 1 4]ODCS/ !  %.}o><46 S/ !%+ /- /!1cg)z$Split text into multiple components.N)r$texts r, split_textTextSplitter.split_textIsr/Nc U=(d 0/[U5-n/n[U5HupVSnSnURU5Hn [R"X55n UR (a<Xx-UR - n URU [SU 55nXzS'[U 5n[XS9n URU 5 M M U$)z&Create documents from a list of texts.r start_index) page_contentmetadata) len enumerater3copydeepcopyr"rfindmaxrappend) r$texts metadatas _metadatas documentsir2indexprevious_chunk_lenchunkr8offsetnew_docs r,create_documentsTextSplitter.create_documentsMs32$U"3   'GAE!" .==7(("7$:M:MMF IIeSF^>B>N>N=OQ{#a'//+AC C( "5"55 [9IA9M STU**+!AI!6!6{1~!F-0-=-AMq"'2!"o  "5"55 [9IA9M STU**+!AI   q ! c+.>.B]J JE9:ook5 ? KK  r/c ^SSKJn [TU5(d Sn[U5eSU4SjjnU"S SU0UD6$![anSn[U5UeSnAff=f) z>Text splitter that uses HuggingFace tokenizer to count length.r)PreTrainedTokenizerBasezATokenizer received was not an instance of PreTrainedTokenizerBasec8>[TRU55$N)r9tokenizer2 tokenizers r,_huggingface_tokenizer_lengthNTextSplitter.from_huggingface_tokenizer.._huggingface_tokenizer_lengths9--d344r/z`Could not import transformers python package. Please install it with `pip install transformers`.Nr'r2strreturnintr1)$transformers.tokenization_utils_basere isinstancer ImportError)clsrjkwargsrer+rkerrs ` r,from_huggingface_tokenizer'TextSplitter.from_huggingface_tokenizerss + Ti)@AAW!o% 5K#@KFKK  +E S/s *  +s,9 AAAgpt2allc  ^^^ SSKnUbURU5m OURU5m SUUU 4Sjjn [ U[ 5(a UUTTS.n 0UEU EnU"SSU 0UD6$![anSn[U5UeSnAff=f) z9Text splitter that uses tiktoken encoder to count length.rNzCould not import tiktoken python package. This is needed in order to calculate max_tokens_for_prompt. Please install it with `pip install tiktoken`.c8>[TRUTTS95$N)allowed_specialdisallowed_special)r9encode)r2r~rencs r,_tiktoken_encoder=TextSplitter.from_tiktoken_encoder.._tiktoken_encoders* $3'9 r/) encoding_name model_namer~rr'rmr1)tiktokenrsencoding_for_model get_encoding issubclassTokenTextSplitter) rtrrr~rrurrvr+r extra_kwargsrs `` @r,from_tiktoken_encoder"TextSplitter.from_tiktoken_encoders ,   !--j9C'' 6C   c, - -!.(#2&8 L 0/,/F?#4???? ,A  c" +  ,sA%% B/A==Bc 6UR[U55$)z2Transform sequence of documents by splitting them.)rNlist)r$rCrus r,transform_documents TextSplitter.transform_documentss##DO44r/)r"rrr!r r#)r%rpr&rpr'zCallable[[str], int]r(z$Union[bool, Literal['start', 'end']]r)boolr*rroNoner2rnro list[str]rg)r@rrAzOptional[list[dict[Any, Any]]]rolist[Document])rCzIterable[Document]ror)rTrrUrnro Optional[str])r\z Iterable[str]rUrnror)rjr rur ror) rrnrrr~'Union[Literal['all'], AbstractSet[str]]r&Union[Literal['all'], Collection[str]]rur ror)rCSequence[Document]rur ror)__name__ __module__ __qualname____firstlineno____doc__r9r-rr3rJrNrVrb classmethodrwsetrr__static_attributes__r1r/r,rrsH3 03?D %!%&2&2&2. &2 = &2  &2&2 &2P33MQ+I (A(TLL,$$(CF5EJ *@*@"*@A *@ C *@  *@ *@*@X5+57:5 5r/cf^\rSrSrSrSS\"5S4S U4SjjjrS SjrSrU=r $) rz/Splitting text to tokens using model tokenizer.ryNrzc >[T U]"S0UD6 SSKnUbUR U5n OUR U5n XlX0lX@lg![anSn[U5UeSnAff=f)zCreate a new TextSplitter.rNzCould not import tiktoken python package. This is needed in order to for TokenTextSplitter. Please install it with `pip install tiktoken`.r1) superr-rrsrr _tokenizer_allowed_special_disallowed_special) r$rrr~rrurrvr+r __class__s r,r-TokenTextSplitter.__init__s "6" ,   !--j9C'' 6C /#5  ,A  c" +  ,sA A-A((A-c^SU4Sjjn[TRTRTRRUS9n[ XS9$)a/Splits the input text into smaller chunks based on tokenization. This method uses a custom tokenizer configuration to encode the input text into tokens, processes the tokens in chunks of a specified size with overlap, and decodes them back into text chunks. The splitting is performed using the `split_text_on_tokens` function. Args: text (str): The input text to be split into smaller chunks. Returns: List[str]: A list of text chunks, where each chunk is derived from a portion of the input text based on the tokenization and chunking rules. cb>TRRUTRTRS9$r})rrrr)_textr$s r,_encode-TokenTextSplitter.split_text.._encodes4??)) $ 5 5#'#;#;* r/)r&tokens_per_chunkdecoderri)rrnroz list[int]) Tokenizerrrrrsplit_text_on_tokens)r$r2rrjs` r,r3TokenTextSplitter.split_textsC  --!--??))  $CCr/)rrr) rrnrrr~rrrrur rorr) rrrrrrr-r3r __classcell__)rs@r,rrsh9$$(CF5EJ 66"6A 6 C 6  6 668DDr/rc\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSr g) Languagei&z"Enum of the programming languages.cppgojavakotlinjstsphpprotopythonrstrubyrustscalaswiftmarkdownlatexhtmlsolcsharpcobolcluaperlhaskellelixir powershell visualbasic6r1N)!rrrrrCPPGOJAVAKOTLINJSrPHPPROTOPYTHONRSTRUBYRUSTSCALASWIFTMARKDOWNLATEXHTMLSOLCSHARPCOBOLCLUAPERLHASKELLELIXIR POWERSHELL VISUALBASIC6rr1r/r,rr&s, C B D F B B C E F C D D E EH E D C F E A C DG FJ!Lr/rT)frozencH\rSrSr%SrS\S'S\S'S\S'S\S 'S rg ) riFzTokenizer data class.rpr&rzCallable[[list[int]], str]rzCallable[[str], list[int]]rr1N)rrrrr__annotations__rr1r/r,rrFs)*, &&= &&=r/rc/nURU5nSn[XAR-[U55nX4UnU[U5:aUR UR U55 U[U5:XaU$XARUR - - n[XAR-[U55nX4UnU[U5:aMU$)z6Split incoming text and return chunks using tokenizer.r)rminrr9r?rr&)r2rjr\ input_ids start_idxcur_idx chunk_idss r,rrTsF  &II)888#i.IGG,I c)n $ i&&y12 c)n $  M //)2I2III i"<"<rs" #!G">>2   8 $ T(E5*CE5P=D =D@"sD"@ $ > > >r/