UnicodeDecodeError: codec can't decode

Python UnicodeDecodeError is disproportionately a Windows problem because Windows uses cp1252 (Windows-1252) as the default system encoding on Western European and US installs, while most modern text files use UTF-8. When you open('file.txt') without an encoding parameter, Python uses locale.getpreferredencoding(False) — which returns cp1252 on most Windows systems and UTF-8 on Linux/macOS. A UTF-8 file containing the em-dash (—), smart quotes (" "), or other characters above ASCII 127 that are undefined in cp1252 triggers UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position N. The file is perfectly valid UTF-8 but the Windows default codec cannot handle it. Always specify encoding='utf-8' explicitly: open('file.txt', encoding='utf-8') is fully portable. Python 3.15 (planned 2025) changes the default encoding behavior, but until then explicit encoding is required. For files with unknown or mixed encodings, the chardet library detects the encoding from the file's bytes: chardet.detect(Path('file.txt').read_bytes()) returns a confidence score and detected encoding name. For CSV files specifically, pandas.read_csv('file.csv', encoding='utf-8-sig') handles the UTF-8 BOM (byte order mark) that Excel prepends when saving CSV files, which causes UnicodeDecodeError in standard UTF-8 mode because the BOM byte sequence is not valid UTF-8 without the BOM variant.