programming python

5 loại mã thông báo trong python là gì?

Câu trả lời

Đơn vị riêng lẻ nhỏ nhất trong chương trình được gọi là Mã thông báo. Python có các mã thông báo sau

Từ khóa — Ví dụ như nhập, cho, trong, trong khi, v.v.
Định danh — Ví dụ là MyFile, DS, DATE9777, v.v.
Chữ — Ví dụ là "abc", 5, 28. 5, v.v.
Toán tử — Ví dụ là +, -, >, hoặc, v.v.
Dấu câu — ' " # [] v.v.

Đối với hầu hết các ứng dụng, không cần thiết phải lo lắng rõ ràng về

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

0, vì

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 dừng lặp lại sau khi mã thông báo cuối cùng được tạo ra

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Loại mã thông báo

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 được sử dụng cho bất kỳ mã định danh Python nào, cũng như mọi từ khóa. Từ khóa là tên Python được đặt trước, nghĩa là chúng không thể được gán cho, chẳng hạn như

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

4 và

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Để biết mã thông báo

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 có phải là từ khóa hay không, hãy sử dụng

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

7 trên

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Như một lưu ý phụ, trong nội bộ, mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32 sử dụng phương pháp

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

50 để kiểm tra xem mã thông báo có phải là mã thông báo

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 hay không. Các quy tắc đầy đủ để tạo nên một mã định danh hợp lệ hơi phức tạp, vì chúng liên quan đến một bảng lớn các ký tự Unicode. Người ta phải luôn sử dụng phương pháp

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

50 để kiểm tra xem một chuỗi có phải là mã định danh Python hợp lệ hay không, kết hợp với kiểm tra

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

7. Việc kiểm tra xem một chuỗi có phải là mã định danh hay không bằng cách sử dụng các biểu thức chính quy rất không được khuyến khích

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

54#

Loại mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

54 được sử dụng cho bất kỳ chữ số nào, bao gồm chữ số nguyên [thập phân], chữ số nguyên nhị phân, bát phân và thập lục phân, số dấu phẩy động [bao gồm ký hiệu khoa học] và chữ số ảo [như

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

56]

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Lưu ý rằng mặc dù các chữ như

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

57 là một loại

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

58 duy nhất, nhưng chúng mã hóa thành

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

54 [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

700],

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701 [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

702],

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

54 [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

704]

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Chữ số không hợp lệ có thể mã hóa dưới dạng nhiều chữ số

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Một lợi thế của việc sử dụng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32 so với

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

706 là các số dấu phẩy động không được làm tròn ở giai đoạn mã thông báo, vì vậy có thể truy cập toàn bộ thông tin đầu vào

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Điều này có thể được sử dụng, ví dụ, để bọc các số dấu phẩy động với một loại hỗ trợ độ chính xác tùy ý, chẳng hạn như

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

707. Xem ví dụ .

Trong Python >=3. 6, chữ số có thể có dấu phân cách gạch dưới, như

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

708

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Trong Trăn 3. 5, điều này sẽ mã hóa dưới dạng hai mã thông báo,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

54 [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

710] và

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

712] [và sẽ không hợp lệ về mặt cú pháp trong bất kỳ ngữ cảnh nào]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Trong ví dụ chúng ta sẽ xem cách sử dụng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32 để nhập tính năng này vào Python 3. 5.

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

714#

Loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

714 khớp với bất kỳ chuỗi ký tự nào, bao gồm chuỗi trích dẫn đơn, chuỗi trích dẫn kép, chuỗi trích dẫn ba đơn và kép [i. e. , chuỗi nhiều dòng hoặc “docstrings”], raw, “unicode”, byte và f-strings [Python 3. 6+]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Lưu ý rằng mặc dù Python hoàn toàn nối các chuỗi ký tự, nhưng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32 mã hóa chúng một cách riêng biệt

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Trong trường hợp chuỗi thô, “unicode”, byte và chuỗi f, tiền tố chuỗi được bao gồm trong chuỗi được mã hóa

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

chuỗi f [Python 3. 6+] được phân tích thành một mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

714 duy nhất

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Hàm

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

718 có thể được sử dụng để phân tích các chuỗi định dạng [bao gồm cả chuỗi f]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Để lấy giá trị chuỗi từ một chuỗi ký tự được mã hóa [i. e. , để loại bỏ các ký tự trích dẫn], hãy sử dụng

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

719. Điều này được khuyến nghị hơn là cố gắng loại bỏ các dấu ngoặc kép theo cách thủ công, dễ bị lỗi hoặc sử dụng

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

720 thô, có thể thực thi mã tùy ý trong trường hợp chuỗi f

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Hành vi lỗi#

Nếu một chuỗi trích dẫn duy nhất không được mở, dấu phân cách chuỗi mở được mã hóa thành ____2721 và phần còn lại được mã hóa như thể nó không .

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Hành vi này có thể hữu ích để xử lý các tình huống lỗi. Ví dụ: nếu bạn định xây dựng một công cụ đánh dấu cú pháp bằng cách sử dụng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32, bạn có thể không nhất thiết muốn một chuỗi không đóng để đánh dấu phần còn lại của tài liệu dưới dạng một chuỗi [những thứ như vậy phổ biến trong môi trường chỉnh sửa “trực tiếp”]

Tuy nhiên, nếu một chuỗi trích dẫn ba [i. e. , chuỗi nhiều dòng hoặc “docstring”] không được đóng,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32 sẽ tăng

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

724 khi đạt đến mức đó.

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Hành vi này có thể khó xử lý trong thực tế. Đối với nhiều ứng dụng, cách chính xác để xử lý tình huống này là xem xét rằng vì chuỗi không được đóng là nhiều dòng nên phần còn lại của đầu vào từ đó ____2724 is raised is inside the unclosed string.

Lưu ý cuối cùng, hãy lưu ý rằng có thể xây dựng các chuỗi ký tự mã hóa mà không có bất kỳ lỗi nào, nhưng tăng

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

726 khi được trình thông dịch phân tích cú pháp

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Để kiểm tra xem một chuỗi ký tự có hợp lệ hay không, bạn có thể sử dụng hàm

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

719, đây là hàm an toàn để sử dụng cho đầu vào không đáng tin cậy

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

728#

Loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

728 đại diện cho một ký tự xuống dòng [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

730 hoặc

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

731] kết thúc một dòng logic của mã Python. Các dòng mới không kết thúc một dòng logic của mã Python sử dụng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

36 .

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Các dòng mới kiểu Windows [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

731] được mã hóa thành một mã thông báo duy nhất

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Bắt đầu từ Python 3. 6. 7 và 3. 7. 1 phát hành bản vá,

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

728 được phát ra ở cuối tất cả đầu vào tokenize ngay cả khi nó không kết thúc bằng một dòng mới [ngoại trừ các dòng chỉ là nhận xét]. Thay đổi này được thực hiện để thống nhất với tokenize nội bộ. mô-đun c được sử dụng bởi chính Python. Các

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

728 ngầm định có

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

3 được đặt thành

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

4. Thay đổi này không được thực hiện đối với Python 3. 5, đã có trong "chỉ sửa lỗi bảo mật". Các ví dụ trong tài liệu này đều sử dụng 3. 6. 7+ hành vi. Nếu muốn nhất quán, người ta luôn có thể buộc đầu vào kết thúc bằng một dòng mới [đây là lý do tại sao mọi ví dụ trong tài liệu này đều có một dòng mới ở cuối]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738#

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739#

Loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 đại diện cho thụt đầu dòng cho các khối được thụt lề. Bản thân phần thụt lề [văn bản từ đầu dòng đến ký tự không phải khoảng trắng đầu tiên] nằm trong thuộc tính

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 được phát ra một lần trên mỗi khối văn bản thụt lề, không phải một lần trên mỗi dòng

Loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739 đại diện cho một vết lõm. Mỗi mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 được khớp với mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739 tương ứng. Thuộc tính

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

2 của

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739 luôn là

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

4. Vị trí

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

5 và

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

6 của mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739 là vị trí đầu tiên trong dòng sau dấu thụt [ngay cả khi có nhiều

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739 liên tiếp]

Xét ví dụ giả sau

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Có một

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 trước khối

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

754, một

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 trước

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

756 và hai

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

757 trước

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

758

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 không được sử dụng để thụt lề trên phần tiếp theo của dòng

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Thụt đầu dòng có thể là số lượng khoảng trắng hoặc tab bất kỳ. Hạn chế duy nhất là mọi mức thụt lề không thụt vào phải khớp với mức thụt lề ngoài trước đó. Nếu thụt lề không khớp với mức thụt lề ngoài, thì

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 tăng ______2761 .

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Mức độ thụt đầu dòng tại một điểm cụ thể trong luồng mã thông báo có thể được xác định bằng cách tăng và giảm bộ đếm cho từng mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

739 hoặc nếu yêu cầu khoảng cách thụt đầu dòng chính xác, bằng cách duy trì một ngăn xếp chuỗi

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

738 . Xem ví dụ .

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

765#

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

766#

Mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

765 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

766 được mã hóa thành

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701 . Tuy nhiên, do một lỗi xuất hiện trong các phiên bản Python trước 3. 7, thuộc tính

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

770 của các mã thông báo này sẽ là

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701 thay vì loại chính xác.

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Lỗi này đã được sửa trong Python 3. 7

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701#

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701 là loại mã thông báo chung cho tất cả toán tử, dấu phân cách và dấu chấm lửng bằng chữ. Điều này không bao gồm các ký tự và toán tử không được trình phân tích cú pháp nhận dạng [chúng được phân tích thành

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721 ].

Khi sử dụng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1, loại mã thông báo cho một toán tử, dấu phân cách hoặc mã thông báo bằng dấu chấm lửng sẽ là

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701. Để có được loại mã thông báo chính xác, hãy sử dụng thuộc tính

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

770 của têntuple.

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

778 tương đương với

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

779 đối với các loại mã thông báo còn lại [với hai trường hợp ngoại lệ, xem ghi chú bên dưới]

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Bảng sau liệt kê tất cả các loại

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

701 chính xác và các ký tự tương ứng của chúng

Loại mã thông báo chính xác

Chuỗi giá trị

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

781

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

782

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

783

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

784

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

785

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

786

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

787

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

788

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

789

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

790

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

791

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

792

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

793

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

794

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

795

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

702

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

797

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

798

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

799

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

765 [ 1 ]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

766 [ 1 ]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

71 [ 2 ]

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

73#

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

74#

Các loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

73 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

74 được sử dụng để mã hóa các từ khóa

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

77 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 trong Python 3. 5 và 3. 6. Chúng không tồn tại trong Python 3. 7+

Trong Trăn 3. 5 và 3. 6,

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

77 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 là từ khóa giả. Để hỗ trợ quá trình chuyển đổi trong việc bổ sung các từ khóa mới,

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

77 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 được giữ làm tên biến hợp lệ bên ngoài khối

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Để hỗ trợ điều này, trong Python 3. 5 và 3. 6 khi gặp phải

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

83,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32 sẽ theo dõi mức thụt đầu dòng của nó và tất cả mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

77 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 được lồng bên dưới nó được mã hóa lần lượt là

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

73 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

74 [bao gồm cả

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 từ

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

83]. Mặt khác,

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

77 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 được mã hóa thành

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1, như trong ví dụ trên

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Trong Python 3. 7,

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

78 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

77 là các từ khóa thích hợp và được mã hóa thành

>>> print_tokens['1+2j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 0], end=[1, 1], line='1+2j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 1], end=[1, 2], line='1+2j\n']
TokenInfo[type=2 [NUMBER], string='2j', start=[1, 2], end=[1, 4], line='1+2j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 4], end=[1, 5], line='1+2j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1 giống như tất cả các từ khóa khác. Trong Trăn 3. 7, các loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

73 và

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

74 đã bị xóa khỏi mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

38.

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Trong Trăn 3. 8, mã thông báo ________ 274 và ________ 273 đã được thêm lại vào mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

38, nhưng chúng không được mã hóa theo mặc định [hành vi giống như trong 3. 7]. Họ chỉ tạo điều kiện thuận lợi cho cờ

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

04 mới thành

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

05 cho phép phân tích cú pháp Python như các phiên bản cũ hơn

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

06#

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

07#

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

06 và

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

07 được bao gồm ở đây cho đầy đủ, vì chúng nằm trong

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

31 từ điển. Chúng được sử dụng trong mã thông báo C để mã hóa nhận xét loại, nhưng mã thông báo Python chưa mã hóa chúng. Điều này có lẽ sẽ thay đổi trong tương lai, vì mã thông báo Python thường được tạo để có hành vi giống như mã thông báo C. Nó đã được thêm vào Python 3. 8.

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

11#

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

11 cũng được bao gồm ở đây cho đầy đủ. Nó có trong

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

31 từ điển, nhưng nó không thực sự được sử dụng ở bất kỳ đâu trong mô-đun tokenize và không thể được phát ra dưới dạng mã thông báo từ

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1. Nó được sử dụng trong mã thông báo C như một phần của trình phân tích cú pháp PEG để xử lý “từ khóa mềm” như

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

15 và

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

16 chỉ là từ khóa khi được sử dụng trong các ngữ cảnh nhất định. Nó đã được thêm vào Python 3. 10.

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721#

Loại

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721 được sử dụng cho bất kỳ ký tự nào không được nhận dạng. Các đầu vào mã hóa mã hóa

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721 không thể là Python hợp lệ, nhưng loại mã thông báo này được sử dụng để các ứng dụng xử lý mã thông báo có thể thực hiện khôi phục lỗi, vì phần còn lại của luồng đầu vào được mã hóa bình thường. Nó cũng có thể được sử dụng để xử lý các phần mở rộng cho cú pháp Python [xem ví dụ ]. Mỗi ký tự không được nhận dạng được mã hóa riêng.

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721 cũng được sử dụng cho các chuỗi trích dẫn đơn không được đóng trước một dòng mới. Xem phần ____2714 để biết thêm thông tin.

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Nếu chuỗi được tiếp tục và không được đóng lại, toàn bộ chuỗi sẽ được mã hóa dưới dạng mã thông báo lỗi. Mặt khác, chỉ dấu phân cách trích dẫn bắt đầu là

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Trong trường hợp các chuỗi trích dẫn đơn lẻ không được đóng tiếp tục, các khoảng trắng trước chuỗi cũng được mã hóa là

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Điều này không áp dụng cho các chuỗi tiếp tục không được đóng

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Do đó, mã xử lý

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

721 cụ thể cho các chuỗi không được đóng nên kiểm tra

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

35#

Loại mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

35 đại diện cho nhận xét. Nếu một nhận xét kéo dài trên nhiều dòng, thì mỗi dòng sẽ được mã hóa riêng

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Loại mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

35 chỉ tồn tại trong thư viện chuẩn Triển khai Python của

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32. Việc triển khai C được trình thông dịch sử dụng bỏ qua các nhận xét. Trong các phiên bản Python trước 3. 7,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

35 chỉ có thể nhập được từ mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32. Trong 3. 7, nó cũng được thêm vào mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

36#

Loại mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

36 đại diện cho các ký tự dòng mới [

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

730 hoặc

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

731] không kết thúc các dòng mã logic. Các dòng mới kết thúc các dòng logic của vùng mã Python được mã hóa bằng cách sử dụng loại mã thông báo

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

728 .

Có hai trường hợp dòng mới được mã hóa là

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Các dòng mới kết thúc các dòng được tiếp tục sau dấu ngoặc nhọn

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Dòng mới kết thúc dòng trống hoặc dòng chỉ có nhận xét

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Lưu ý rằng các dòng mới được thoát [trước

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

38] được coi là khoảng trắng, nghĩa là chúng hoàn toàn không mã hóa. Do đó, bạn phải luôn sử dụng số dòng trong thuộc tính

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

5 và

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

6 của bộ tên

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

41. Đừng bao giờ thử xác định số dòng bằng cách đếm

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

728 và

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

36 mã thông báo.

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Loại mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

36 chỉ tồn tại trong thư viện chuẩn Triển khai Python của

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32. Việc triển khai C được trình thông dịch sử dụng chỉ có ____2728 . Trong các phiên bản Python trước 3. 7,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

36 chỉ có thể nhập được từ mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32. Trong 3. 7, nó cũng được thêm vào mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

38.

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37#

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 là một loại mã thông báo đặc biệt đại diện cho việc mã hóa đầu vào. Nó luôn là mã thông báo đầu tiên được phát ra bởi

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

1, trừ khi mã hóa được phát hiện không hợp lệ, trong trường hợp đó, nó sẽ tăng

>>> print_tokens['x + 1\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='x', start=[1, 0], end=[1, 1], line='x + 1\n']
TokenInfo[type=54 [OP], string='+', start=[1, 2], end=[1, 3], line='x + 1\n']
TokenInfo[type=2 [NUMBER], string='1', start=[1, 4], end=[1, 5], line='x + 1\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 5], end=[1, 6], line='x + 1\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

726 . Mã hóa được phát hiện thông qua nhận xét có định dạng PEP 263 ở một trong hai dòng đầu tiên của đầu vào [chẳng hạn như

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

54; những nhận xét như vậy vẫn được mã hóa dưới dạng

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

35 as well], or a Unicode BOM character.

Mã hóa được phát hiện nằm trong thuộc tính

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

2 của

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

41.

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 là loại mã thông báo duy nhất mà

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

59 không xuất hiện theo nghĩa đen trong đầu vào. Mã hóa mặc định là

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

Nếu bạn chỉ muốn phát hiện mã hóa và không muốn phát hiện gì khác, hãy sử dụng ____361 . Nếu bạn chỉ cần mã hóa để chuyển tới

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

62, hãy sử dụng

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

63 .

Số dòng và cột của

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

5 và

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

6 cho

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 sẽ luôn là

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

>>> print_tokens['a or α\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=1 [NAME], string='a', start=[1, 0], end=[1, 1], line='a or α\n']
TokenInfo[type=1 [NAME], string='or', start=[1, 2], end=[1, 4], line='a or α\n']
TokenInfo[type=1 [NAME], string='α', start=[1, 5], end=[1, 6], line='a or α\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 6], end=[1, 7], line='a or α\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 thường không được sử dụng khi xử lý mã thông báo, mặc dù sự hiện diện được đảm bảo của nó với tư cách là mã thông báo đầu tiên có thể hữu ích khi xử lý mã thông báo vì nó sẽ đảm bảo rằng mọi loại mã thông báo khác sẽ có một số mã thông báo trước đó từ luồng [xem phần . examples].

Nói một cách chính xác,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

2 của mọi mã thông báo trong luồng mã thông báo phải được giải mã bằng cách mã hóa mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 [e. g. , nếu mã hóa là

>>> print_tokens['']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']

71, mã thông báo không thể bao gồm bất kỳ ký tự không phải ASCII nào]

Loại mã thông báo

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 chỉ tồn tại trong thư viện chuẩn Triển khai Python của

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32. Việc triển khai C được trình thông dịch sử dụng sẽ phát hiện mã hóa riêng. Trong các phiên bản Python trước 3. 7,

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

37 chỉ có thể nhập được từ mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

32. Trong 3. 7, nó cũng được thêm vào mô-đun

>>> print_tokens['10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line='']
TokenInfo[type=2 [NUMBER], string='10', start=[1, 0], end=[1, 2], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 3], end=[1, 4], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0b101', start=[1, 5], end=[1, 10], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 11], end=[1, 12], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0o10', start=[1, 13], end=[1, 17], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 18], end=[1, 19], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='0xa', start=[1, 20], end=[1, 23], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='-', start=[1, 24], end=[1, 25], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1.0', start=[1, 26], end=[1, 29], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 30], end=[1, 31], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1e1', start=[1, 32], end=[1, 35], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=54 [OP], string='+', start=[1, 36], end=[1, 37], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=2 [NUMBER], string='1j', start=[1, 38], end=[1, 40], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=4 [NEWLINE], string='\n', start=[1, 40], end=[1, 41], line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n']
TokenInfo[type=0 [ENDMARKER], string='', start=[2, 0], end=[2, 0], line='']

Có bao nhiêu loại mã thông báo trong Python?

Có 5 các loại mã thông báo trong python được liệt kê bên dưới. từ khóa.

Các loại mã thông báo trong Python là gì?

Python ngắt từng dòng logic thành một chuỗi các thành phần từ vựng cơ bản được gọi là mã thông báo. Mỗi mã thông báo tương ứng với một chuỗi con của dòng logic. Các loại mã thông báo thông thường là mã định danh, từ khóa, toán tử, dấu phân cách và ký tự, như được đề cập trong các phần sau

Các loại mã thông báo khác nhau * là gì?

Các loại mã thông báo .

mã thông báo tiện ích

Mã thông báo hàng hóa

Mã thông báo bảo mật

Mã thông báo nội dung

Các ví dụ về mã thông báo là gì?

Nói chung, mã thông báo là một đối tượng đại diện cho một thứ khác, chẳng hạn như một đối tượng khác [thực tế hoặc ảo] hoặc một khái niệm trừu tượng, chẳng hạn như một món quà< . is sometimes referred to as a token of the giver's esteem for the recipient.

Câu trả lời

Hành vi lỗi#

>>> print_tokens[''] TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line=''] TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line=''] 06#

>>> print_tokens[''] TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line=''] TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line=''] 07#

>>> print_tokens[''] TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line=''] TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line=''] 11#

Có bao nhiêu loại mã thông báo trong Python?

Các loại mã thông báo trong Python là gì?

Các loại mã thông báo khác nhau * là gì?

Các ví dụ về mã thông báo là gì?

Bài Viết Liên Quan

Toplist mới

Bài mới nhất

Chủ Đề

>>> print_tokens[''] TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line=''] TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']
06#

>>> print_tokens[''] TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line=''] TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']
07#

>>> print_tokens[''] TokenInfo[type=63 [ENCODING], string='utf-8', start=[0, 0], end=[0, 0], line=''] TokenInfo[type=0 [ENDMARKER], string='', start=[1, 0], end=[1, 0], line='']
11#