a �DOge�@s>dZdZddlmZddlmZmZddlZddl m Z ddl Z ddl Z ddlZddlTddlmZe �d e j�Ze �d e j�ZddlZejgd �Z[Gd d �d e�d d��Zdd�Zdd�Zdd�ZdZdZeede�ee�ZdZdZdZ dZ!dZ"eee e!e"�Z#dZ$edd�ee$�Z%d e$Z&ee%e&�Z'ed!e'd"�Z(ee(e'e#�Z)d#d$�Z*d%d&�Z+ee*��Z,d'Z-d(Z.d)Z/d*Z0ee,d+e,d,�Z1ee,d-e,d.�Z2ee3e j4e5ed/d0���Z6ed1e6�Z7ee)e7e2e�Z8ee8Z9ee,d2ed3d�e,d4ed5d��Z:ed6ee1�Z;eee;e)e7e:e�Ze-e=e>d3<e.e=e>d5<e/e=e>d+<e0e=e>d,<�q"e?�Z@e?�ZAe*�D]JZBeBd5eBd3fD]ZCe@�DeC��q�eBd,eBd+fD]ZCeA�DeC��q��qld7ZEGd8d9�d9eF�ZGGd:d;�d;eF�ZHGdd?�ZJd@dA�ZKdBdC�ZLdDdE�ZdFdG�ZMdHdI�ZNdJdK�ZOdLdM�ZPeQdNk�r:eP�dS)OzKa-Ping Yee zpGvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro, Raymond Hettinger, Trent Nelson, Michael Foord�)�open)�lookup�BOM_UTF8N)� TextIOWrapper)�*)�EXACT_TOKEN_TYPESz&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$))�tokenize�generate_tokens�detect_encoding� untokenize� TokenInfoc@s eZdZdd�Zedd��ZdS)r cCs$d|jt|jf}d|j|d�S)Nz%d (%s)z8TokenInfo(type=%s, string=%r, start=%r, end=%r, line=%r))�type)r �tok_name�_replace)�self�annotated_type�r� /usr/lib64/python3.9/tokenize.py�__repr__.s �zTokenInfo.__repr__cCs(|jtkr|jtvrt|jS|jSdS�N)r �OP�stringr�rrrr� exact_type3s zTokenInfo.exact_typeN)�__name__� __module__� __qualname__r�propertyrrrrrr -sr ztype string start end linecGsdd�|�dS)N�(�|�))�join��choicesrrr�group:�r$cGs t|�dS)Nr�r$r"rrr�any;r%r'cGs t|�dS)N�?r&r"rrr�maybe<r%r)z[ \f\t]*z #[^\r\n]*z\\\r?\nz\w+z0[xX](?:_?[0-9a-fA-F])+z0[bB](?:_?[01])+z0[oO](?:_?[0-7])+z(?:0(?:_?0)*|[1-9](?:_?[0-9])*)z[eE][-+]?[0-9](?:_?[0-9])*z)[0-9](?:_?[0-9])*\.(?:[0-9](?:_?[0-9])*)?z\.[0-9](?:_?[0-9])*z[0-9](?:_?[0-9])*z[0-9](?:_?[0-9])*[jJ]z[jJ]cCsVgd�}dh}|D]>}t�|�D].}tjdd�|D��D]}|�d�|��q8q q|S)N)�b�r�u�f�br�fr�cSsg|]}||��f�qSr)�upper)�.0�crrr� ^r%z(_all_string_prefixes..)� _itertools� permutations�product�addr!)�_valid_string_prefixes�result�prefix�tr,rrr�_all_string_prefixesSsr=cCst�|tj�Sr)�re�compile�UNICODE)�exprrrr�_compilebsrBz[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z'''z"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"T)�reversez\r?\nz'[^\n'\\]*(?:\\.[^\n'\\]*)*�'z"[^\n"\\]*(?:\\.[^\n"\\]*)*�"z \\\r?\n|\Z�c@s eZdZdS)� TokenErrorN�rrrrrrrrG�r%rGc@s eZdZdS)�StopTokenizingNrHrrrrrI�r%rIc@s,eZdZdd�Zdd�Zdd�Zdd�Zd S) � UntokenizercCsg|_d|_d|_d|_dS)N�r)�tokens�prev_row�prev_col�encodingrrrr�__init__�szUntokenizer.__init__cCs�|\}}||jks&||jkr>||jkr>td�|||j|j���||j}|rb|j�d|�d|_||j}|r�|j�d|�dS)Nz+start ({},{}) precedes previous end ({},{})�\ r� )rMrN� ValueError�formatrL�append)r�start�row�col� row_offset� col_offsetrrr�add_whitespace�s �  zUntokenizer.add_whitespacec Cs6t|�}g}d}|D�]}t|�dkr8|�||��q*|\}}}} } |tkrV||_q|tkrd�q*|tkrz|�|�qnl|tkr�|� �| \|_ |_ qnL|t t fvr�d}n:|r�|r�|d} |dt| �kr�|j�| �t| �|_ d}|�|�|j�|�| \|_ |_ |t t fvr|j d7_ d|_ qd�|j�S)NF�T�����rKrr0)�iter�len�compat�ENCODINGrO� ENDMARKER�INDENTrU�DEDENT�poprMrN�NEWLINE�NLrLr[r!) r�iterable�it�indents� startliner<�tok_type�tokenrV�end�line�indentrrrr �sF            zUntokenizer.untokenizec Cs�g}|jj}|dttfv}d}t�|g|�D]�}|dd�\}} |tkrR| |_q.|tt fvrf| d7} |t kr�|rzd| } d}nd}|t kr�|�| �q.n>|t kr�|� �q.n*|ttfvr�d}n|r�|r�||d�d}|| �q.dS)NrFr\rRTr])rLrUrfrgr5�chainrarO�NAME�NUMBER�STRINGrcrdre) rrmrhrj� toks_appendrk� prevstring�tok�toknum�tokvalrrrr`�s8    zUntokenizer.compatN)rrrrPr[r r`rrrrrJ�s %rJcCs*t�}|�|�}|jdur&|�|j�}|Sr)rJr rO�encode)rh�ut�outrrrr s    r cCsH|dd����dd�}|dks*|�d�r.dS|dvs@|�d�rDdS|S) N� �_�-�utf-8zutf-8-)zlatin-1� iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-r�)�lower�replace� startswith)�orig_enc�encrrr�_get_normal_names�r�cs�z �jj�Wnty"d�Yn0d�d}d}�fdd�}��fdd�}|�}|�t�rnd�|dd�}d }|sz|gfS||�}|r�||gfSt�|�s�||gfS|�}|s�||gfS||�}|r�|||gfS|||gfS) NFr�cs"z��WStyYdS0dS�Nr%)� StopIterationr��readlinerr� read_or_stop?s z%detect_encoding..read_or_stopcs�z|�d�}Wn2ty@d}�dur4d�|��}t|��Yn0t�|�}|sTdSt|�d��}z t|�}Wn8t y��dur�d|}n d��|�}t|��Yn0�r�|dkrԈdur�d}n d���}t|��|d 7}|S) Nr�z'invalid or missing encoding declarationz {} for {!r}rKzunknown encoding: zunknown encoding for {!r}: {}zencoding problem: utf-8z encoding problem for {!r}: utf-8z-sig) �decode�UnicodeDecodeErrorrT� SyntaxError� cookie_re�matchr�r$r� LookupError)ro� line_string�msgr�rO�codec)� bom_found�filenamerr� find_cookieEs8      � z$detect_encoding..find_cookieT�� utf-8-sig)�__self__�name�AttributeErrorr�r�blank_rer�)r�rO�defaultr�r��first�secondr)r�r�r�rr 's8    &       r cCsXt|d�}z2t|j�\}}|�d�t||dd�}d|_|WS|���Yn0dS)N�rbrT)�line_bufferingr+)� _builtin_openr r��seekr�mode�close)r��bufferrO�lines�textrrrr�s  rcCs6t|�\}}t�d�}t�|t|d�|�}t|j|�Sr�)r r5�repeatrqr^� _tokenize�__next__)r�rO�consumed�empty�rl_genrrrr�s  rccszd}}}d}d\}}d}dg} |durH|dkr6d}tt|ddd�Vd} d} z| } |�} Wntytd} Yn0|dur�| �|�} |d 7}dt| �} } |�rn| s�td |��|�| �}|�r|�d�} }tt|| d|�|||f|| �Vd\}}d}nf|�rX| d d�d k�rX| d d�dk�rXtt || ||t| �f|�Vd}d}qPn|| }|| }qP�n�|dk�r|�s| �s��q�d}| | k�r�| | dk�r�|d 7}n8| | dk�r�|t d t }n| | dk�r�d}n�q�| d 7} �q�| | k�r�q�| | dv�r�| | dk�r\| | d�� d�}tt ||| f|| t|�f| �V| t|�7} tt | | d�|| f|t| �f| �VqP|| dk�r�| �|�tt| d| �|df|| f| �V|| dk�r,|| v�r�tdd|| | f��| dd�} ttd|| f|| f| �V�q�n| �s(td|df��d}| | krPtt��| | �}|�r�|�d �\}}||f||f|}}} ||k�r~�q,| ||�| |}}||v�s�|dk�r�|dk�r�|dk�r�tt|||| �V�q�|dv�r|dk�r�tt |||| �Vntt|||| �V�q�|dk�r0tt |||| �V�q�|tv�r�tt|�}|�| | �}|�r�|�d�} | || �}tt|||| f| �Vn||f}| |d�}| }qP�q�|tv�s�|dd�tv�s�|dd�tv�rD|ddk�r0||f}tt�|��pt�|d ��pt�|d��}| |d�d }}| }qPntt|||| �Vnf|���rbtt|||| �VnH|dk�rrd }n8|dv�r�|d 7}n|d v�r�|d 8}tt|||| �Vn*tt | | || f|| d f| �V| d 7} �q,qP| �r.| ddv�r.| ���d��s.ttd|d t| �f|d t| �d fd�V| d d�D] }ttd|df|dfd�V�q:tt d|df|dfd�VdS)!Nr� 0123456789)r0rr�r�)rrr0r%rKzEOF in multi-line string�����rQ�����z\ rR� � z# �#z r]z3unindent does not match any outer indentation levelz zEOF in multi-line statement�.z...r\r�� �\z([{z)]})!r rar�r�r_rGr�rnrt� ERRORTOKEN�tabsize�rstrip�COMMENTrgrUrc�IndentationErrorrdrB� PseudoToken�spanrsrf� triple_quoted�endpats� single_quoted�get� isidentifierrrr�stripr�rb)r�rO�lnum�parenlev� continued�numchars�contstr�needcont�contlinerj� last_linero�pos�max�strstart�endprog�endmatchrn�column� comment_token� pseudomatchrV�spos�eposrm�initialrprrrr��s&       �* �     � � "  �    ���          �� � �      �$.r�cCs t|d�Sr)r�r�rrrr dsr c s@ddl}dd��d�fdd� }|jdd�}|jdd d d d �|jd ddddd�|��}z�|jr�|j}t|d��}tt|j��}Wd�q�1s�0Ynd}t t j jd�}|D]>}|j }|j r�|j}d|j|j} td| t||jf�q�W�n@t�yL} z8| jddd�\} } || jd|| | f�WYd} ~ n�d} ~ 0t�y�} z0| jd\} } || jd|| | f�WYd} ~ n�d} ~ 0t�y�} z|| |�WYd} ~ n�d} ~ 0t�y�} z|| �WYd} ~ nVd} ~ 0t�ytd�Yn4t�y:} z�d| ��WYd} ~ n d} ~ 00dS)NrcSstj�|�tj�d�dS)Nr�)�sys�stderr�write)�messagerrr�perrorps zmain..perrorcsR|r"|f||f}�d|�n"|r8�d||f�n �d|�t�d�dS)Nz%s:%d:%d: error: %sz %s: error: %sz error: %srK)r��exit)r�r��location�args�r�rr�errorts zmain..errorzpython -m tokenize)�progr�r(z filename.pyz'the file to tokenize; defaults to stdin)�dest�nargs�metavar�helpz-ez--exact�exact� store_truez(display token names using the exact type)r��actionr�r�zz %d,%d-%d,%d:z%-20s%-15s%-15rrKr�z interrupted zunexpected error: %s)NN)�argparse�ArgumentParser� add_argument� parse_argsr�r��listrr�r�r��stdinr r�rrVrn�printrrr�r�rGr��OSError�KeyboardInterrupt� Exception) r�r��parserr�r�r-rLrm� token_type� token_range�errror�rr�r�mainlsT � � .�**  r��__main__)R� __author__� __credits__�builtinsrr��codecsrr� collections�ior� itertoolsr5r>r�rmrr?�ASCIIr�r��__all__� namedtupler r$r'r)� Whitespace�Comment�Ignore�Name� Hexnumber� Binnumber� Octnumber� Decnumber� Intnumber�Exponent� Pointfloat�Expfloat� Floatnumber� Imagnumber�Numberr=rB� StringPrefix�Single�Double�Single3�Double3�Triple�String�map�escape�sorted�Special�Funny� PlainToken�Token�ContStr� PseudoExtrasr�r��_prefix�setr�r�r<r,r8r�r�rGrIrJr r�r rr�r r�rrrrr�s�    ��   � ���      _ ]8=