rdZddlmZddlmZddlmZddlZgdZdZGdd eZ Gd d eZ d Z y) z!A utility for tokenizing strings.)absolute_import)division)unicode_literalsN)Literal SeparatorTokenize\c eZdZy)rN__name__ __module__ __qualname__)lib/googlecloudsdk/core/util/tokenizer.pyrrrrc eZdZy)rNr rrrrr!rrrcdg}tj}tj|} |jd}|sn|tk(rQ|jd}|r|j |Ct dj |jdz |||vrW|jt|j|jt|tj}n|j ||jt|j|S)aTokenizes the given string based on a list of separator strings. This is similar to splitting the string based on separators, except that this function retains the separators. The separators are wrapped in Separator objects and everything else is wrapped in Literal objects. For example, Tokenize('a:b,c:d', [':', ',']) returns [Literal('a'), Separator(':'), Literal('b'), Separator(','), Literal('c'), Separator(':'), Literal('d')]. Args: string: str, The string to partition. separators: [str], A list of strings on which to partition. Raises: ValueError: If an unterminated escape sequence is at the end of the input. Returns: [tuple], A list of strings which can be of types Literal or Separator. z)illegal escape sequence at index {0}: {1}) ioStringIOread _ESCAPE_CHARwrite ValueErrorformattellappendrgetvaluer)string separatorstokenscurrbufcs rrr%s2 & $ F#   A  l  ((1+a  1 DKK HHJNF$% % j mmGDMMO,- mmIaL! [[]d jjm# & -- () -r) __doc__ __future__rrrr__all__rstrrrrrrrr*s= (&' . c1r