A Reliable way to detect Japanese ShiftJIS encoded files?
The other day u/Practical_Air6315 had a couple of threads dealing with issues ex: not knowing if a file is ShiftJIS or UTF8NoBOM encoded
Can you just decode as utf8, checking for errors? If yes, use ShiftJIS otherise utf8? Or can you sometimes have zero decoding errors but it still maps to malformed json? Is there a better method?
I used:
function Test-ShiftJISDecodeError {
# ...
$Utf8Strict = [System.Text.UTF8Encoding]::new(
<# shouldEmitUtf8BOM #> $false,
<# should throw on decode error #> $true )
$bytes = [System.IO.File]::ReadAllBytes( $File.FullName )
try {
[void] $Utf8Strict.GetString( $bytes )
return $false
}
catch [System.Text.DecoderFallbackException] {
return $true
}
}
Here's a test file Make-ShiftJISFile.ps1 ( for Win PS 5.1 and 7 )
And another ShiftJIS example: github/donuts: Compare-Encoding-Breaking-Emojibake.md
Posted Aug 25, 2026, 8:48 PM UTC
24