Hướng dẫn read file utf-8 python

You have stumbled over the general problem with encodings: How can I tell in which encoding a file is?

Answer: You can't unless the file format provides for this. XML, for example, begins with:

This header was carefully chosen so that it can be read no matter the encoding. In your case, there is no such hint, hence neither your editor nor Python has any idea what is going on. Therefore, you must use the codecs module and use codecs.open(path,mode,encoding) which provides the missing bit in Python.

As for your editor, you must check if it offers some way to set the encoding of a file.

The point of UTF-8 is to be able to encode 21-bit characters (Unicode) as an 8-bit data stream (because that's the only thing all computers in the world can handle). But since most OSs predate the Unicode era, they don't have suitable tools to attach the encoding information to files on the hard disk.

The next issue is the representation in Python. This is explained perfectly in the comment by heikogerlach. You must understand that your console can only display ASCII. In order to display Unicode or anything >= charcode 128, it must use some means of escaping. In your editor, you must not type the escaped display string but what the string means (in this case, you must enter the umlaut and save the file).

That said, you can use the Python function eval() to turn an escaped string into a string:

>>> x = eval("'Capit\\xc3\\xa1n\\n'")
>>> x
'Capit\xc3\xa1n\n'
>>> x[5]
'\xc3'
>>> len(x[5])
1

As you can see, the string "\xc3" has been turned into a single character. This is now an 8-bit string, UTF-8 encoded. To get Unicode:

>>> x.decode('utf-8')
u'Capit\xe1n\n'

Gregg Lind asked: I think there are some pieces missing here: the file f2 contains: hex:

0000000: 4361 7069 745c 7863 335c 7861 316e  Capit\xc3\xa1n

codecs.open('f2','rb', 'utf-8'), for example, reads them all in a separate chars (expected) Is there any way to write to a file in ASCII that would work?

Answer: That depends on what you mean. ASCII can't represent characters > 127. So you need some way to say "the next few characters mean something special" which is what the sequence "\x" does. It says: The next two characters are the code of a single character. "\u" does the same using four characters to encode Unicode up to 0xFFFF (65535).

So you can't directly write Unicode to ASCII (because ASCII simply doesn't contain the same characters). You can write it as string escapes (as in f2); in this case, the file can be represented as ASCII. Or you can write it as UTF-8, in which case, you need an 8-bit safe stream.

Your solution using decode('string-escape') does work, but you must be aware how much memory you use: Three times the amount of using codecs.open().

Remember that a file is just a sequence of bytes with 8 bits. Neither the bits nor the bytes have a meaning. It's you who says "65 means 'A'". Since \xc3\xa1 should become "à" but the computer has no means to know, you must tell it by specifying the encoding which was used when writing the file.

You have stumbled over the general problem with encodings: How can I tell in which encoding a file is?

Nội dung chính

• Opening and Closing a "File Object"
• Reading from a File
• Writing to a File
• Common Pitfalls
• How do I open a file encoding in Python?
• How do I open a UTF
• Can Python read UTF
• How do you decode a file in Python?

Answer: You can't unless the file format provides for this. XML, for example, begins with:

This header was carefully chosen so that it can be read no matter the encoding. In your case, there is no such hint, hence neither your editor nor Python has any idea what is going on. Therefore, you must use the codecs module and use codecs.open(path,mode,encoding) which provides the missing bit in Python.

As for your editor, you must check if it offers some way to set the encoding of a file.

The point of UTF-8 is to be able to encode 21-bit characters (Unicode) as an 8-bit data stream (because that's the only thing all computers in the world can handle). But since most OSs predate the Unicode era, they don't have suitable tools to attach the encoding information to files on the hard disk.

The next issue is the representation in Python. This is explained perfectly in the comment by heikogerlach. You must understand that your console can only display ASCII. In order to display Unicode or anything >= charcode 128, it must use some means of escaping. In your editor, you must not type the escaped display string but what the string means (in this case, you must enter the umlaut and save the file).

That said, you can use the Python function eval() to turn an escaped string into a string:

>>> x = eval("'Capit\\xc3\\xa1n\\n'")
>>> x
'Capit\xc3\xa1n\n'
>>> x[5]
'\xc3'
>>> len(x[5])
1

As you can see, the string "\xc3" has been turned into a single character. This is now an 8-bit string, UTF-8 encoded. To get Unicode:

>>> x.decode('utf-8')
u'Capit\xe1n\n'

Gregg Lind asked: I think there are some pieces missing here: the file f2 contains: hex:

0000000: 4361 7069 745c 7863 335c 7861 316e  Capit\xc3\xa1n

codecs.open('f2','rb', 'utf-8'), for example, reads them all in a separate chars (expected) Is there any way to write to a file in ASCII that would work?

Answer: That depends on what you mean. ASCII can't represent characters > 127. So you need some way to say "the next few characters mean something special" which is what the sequence "\x" does. It says: The next two characters are the code of a single character. "\u" does the same using four characters to encode Unicode up to 0xFFFF (65535).

So you can't directly write Unicode to ASCII (because ASCII simply doesn't contain the same characters). You can write it as string escapes (as in f2); in this case, the file can be represented as ASCII. Or you can write it as UTF-8, in which case, you need an 8-bit safe stream.

Your solution using decode('string-escape') does work, but you must be aware how much memory you use: Three times the amount of using codecs.open().

Remember that a file is just a sequence of bytes with 8 bits. Neither the bits nor the bytes have a meaning. It's you who says "65 means 'A'". Since \xc3\xa1 should become "à" but the computer has no means to know, you must tell it by specifying the encoding which was used when writing the file.

On this page: open(), file.read(), file.readlines(), file.write(), file.writelines().

Opening and Closing a "File Object"

1. Create a file object using the open() function. Along with the file name, specify:
   • 'r' for reading in an existing file (default; can be dropped),
   • 'w' for creating a new file for writing,
   • 'a' for appending new content to an existing file.
2. Do something with the file object (reading, writing).
3. Close the file object by calling the .close() method on the file object.

myfile = open('alice.txt', 'r')     # Reading. 'r' can be omitted
# ... read from myfile ...
myfile.close()                      # Closing file
foo.py

myfile = open('results.txt', 'w')   # The file is newly created where foo.py is
# ... write to myfile ...
myfile.close()                      # Closing file. VERY IMPORTANT!

myfile = open('results.txt', 'a')   # 'a': appending instead of overwriting.
# ... add text to the file ...
myfile.close()                      # Closing file. DON'T FORGET!
foo.py 

myfile = open('alice.txt', encoding='utf-8')     # Reading a UTF-8 file; 'r' is omitted


myfile = open('results.txt', 'w', encoding='utf-8')   # File will be written in UTF-8

foo.py

Reading from a File

First off,

>>> f = open('mary-short.txt')
>>> marytxt = f.read()             # Using .read()
>>> f.close()
>>> marytxt
'Mary had a little lamb,\nHis fleece was white as snow,\nAnd everywhere that Mary 
went,\nThe lamb was sure to go.\n'
>>> type(marytxt)                  # marytxt is string type

>>> len(marytxt)                   # marytxt has 110 characters
110
>>> print(marytxt[0])
M

>>> f = open('mary-short.txt')
>>> marylines = f.readlines()      # Using .readlines()
>>> f.close()
>>> marylines
['Mary had a little lamb,\n', 'His fleece was white as snow,\n', 'And everywhere 
that Mary went,\n', 'The lamb was sure to go.\n']
>>> type(marylines)                # marylines is list type

>>> len(marylines)                 # marylines has 4 lines
4
>>> print(marylines[0])
Mary had a little lamb,

f = open('bible-kjv.txt')     # This is a big file
for line in f:                # Using 'for ... in' on file object
    if 'smite' in line: 
        print(line,)                # ',' keeps print from adding a line break

f.close()
foo.py

Writing to a File

Below,

>>> fout = open('hello.txt', 'w')
>>> fout.write('Hello, world!\n')                # .write(str)
>>> fout.write('My name is Homer.\n')
>>> fout.write("What a beautiful day we're having.\n")
>>> fout.close()

>>> tobuy = ['milk\n', 'butter\n', 'coffee beans\n', 'arugula\n']
>>> fout = open('grocerylist.txt', 'w')
>>> fout.writelines(tobuy)                      # .writelines(list)
>>> fout.close()

Common Pitfalls

"No such file or directory" error

>>> f = open('mary-short.txt')
Traceback (most recent call last):
  File "", line 1, in 
IOError: [Errno 1] No such file or directory: 'mary-short.txt'

Issues with encoding

>>> f = open('mary-short.txt')    # need encoding='utf-8'
>>> marytxt = f.read()
Traceback (most recent call last):
  File "", line 1, in 
    marytxt = f.read()
  File "C:\Program Files (x86)\Python35-32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 36593: character 
maps to  

Entire file content can be read in only ONCE per opening

>>> f = open('mary-short.txt')
>>> marytxt = f.read()           # Reads in entire file content
>>> marylines = f.readlines()    # Nothing left to read, returns nothing
>>> f.close()
>>> len(marytxt)
110
>>> len(marylines)     # marylines is empty!
0

Only the string type can be written

>>> pi = 3.141592
>>> fout = open('math.txt', 'w')
>>> fout.write("Pi's value is ")
>>> fout.write(pi)                # trying to write float, doesn't work
Traceback (most recent call last):
  File "", line 1, in 
TypeError: expected a character buffer object
>>> fout.write(str(pi))          # turn number into string using str()  
>>> 

Your output file is empty

(Windows) Line breaks do not show up
If you open up your text file in Notepad app in Windows and see everything in one line, don't be alarmed. Open the same text file in Wordpad or, even better, Notepad++, and you will see that the line breaks are there after all. See this FAQ for details.

How do I open a file encoding in Python?

To open a file, you can use Python's built-in open() function. Inside the open() function parentheses, you insert the filepath to be opened in quotation marks. You should also insert a character encoding, which we will talk more about below. This function returns what's called a file object.

How do I open a UTF

How to Open UTF-8 in Excel.

Launch Excel and select "Open Other Workbooks" from the opening screen. ... .

Select "Computer," and then click "Browse." Navigate to the location of the UTF file, and then change the file type option to "All Files.".

Select the UTF file, and then click "Open" to launch the Text Import Wizard..

Can Python read UTF

UTF-8 is one of the most commonly used encodings, and Python often defaults to using it.

How do you decode a file in Python?

decode() is a method specified in Strings in Python 2. This method is used to convert from one encoding scheme, in which argument string is encoded to the desired encoding scheme. This works opposite to the encode. It accepts the encoding of the encoding string to decode it and returns the original string.