SmartOS: Stock "mkvtoolnix" crashes on 2023.4.0 with locale error
This post transcribes some emails I sent to the SmartOS mailing list some time ago. You can check the original thread:
Message-ID: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> Date: Tue, 12 Mar 2024 14:07:13 +0100 To: smartos-discuss <smartos-discuss@lists.smartos.org> From: Jesus Cea <jcea@jcea.es> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> Subject: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error Hi, there. I install "pkgin install mkvtoolnix" with no issues (but a huge number of dependencies, 1.6 Gigabytes total!). Then, I run "mkvmerge" and it crashes: """ [jcea@test ~]$ mkvmerge terminate called after throwing an instance of 'std::runtime_error' what(): locale::facet::_S_create_c_locale name not valid Abort (core dumped) """ This worked fine in 2022.4.0. My environment variables are: """ MANPATH=/opt/local/man:/usr/share/man:/opt/local/lib/perl5/man:/opt/local/lib/perl5/vendor_perl/man LC_MONETARY=es_ES.UTF-8 TERM=xterm SHELL=/usr/bin/bash SSH_CLIENT=*** LC_NUMERIC=es_ES.UTF-8 SSH_TTY=/dev/pts/26 USER=root COLUMNS=80 PAGER=less MAIL=/var/mail/root PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin LC_COLLATE=es_ES.UTF-8 PWD=/home/jcea LANG=en_US.UTF-8 TZ=Europe/Madrid LINES=25 SHLVL=1 HOME=/home/jcea TERMINFO=/opt/local/share/lib/terminfo LOGNAME=jcea SSH_CONNECTION=**** LC_TIME=es_ES.UTF-8 FTPMODE=auto _=/opt/local/bin/env """
Jonathan Perkin replied:
Date: Tue, 12 Mar 2024 15:25:00 +0000 From: Jonathan Perkin <jperkin@mnx.io> To: smartos-discuss <smartos-discuss@lists.smartos.org> Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error Message-ID: <ZfBzzJyHEb-Ujy64@mnx.io> References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> In-Reply-To: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> * On 2024-03-12 at 13:09 GMT, Jesus Cea wrote: >I install "pkgin install mkvtoolnix" with no issues (but a huge number >of dependencies, 1.6 Gigabytes total!). Yeh, mkvtoolnix is not a small command line utility! It is a very large program, with many dependencies, including huge swathes of QT6. >[jcea@test ~]$ mkvmerge >terminate called after throwing an instance of 'std::runtime_error' > what(): locale::facet::_S_create_c_locale name not valid >Abort (core dumped) I've narrowed this down to a smaller test case: --- #include <codecvt> #include <iostream> #include <locale> int main() { auto const &locale_name = "en_US.UTF-8"; auto utf8_locale = std::locale { std::locale{locale_name}, new std::codecvt_utf8<wchar_t> }; try { std::locale::global(utf8_locale); } catch (std::runtime_error &) { } } --- This causes the same exception on us with both GCC 7 and 13 (as it's coming from libstdc++ I tried an older release to rule out GCC 13 related issues), but works fine on NetBSD, macOS, etc. I'll need to dig in further with the --enable-clocale option for libstdc++ to see if we can improve this. Until then you might be able to work around it by unsetting your locale environment variables, i.e.: $ mkvmerge mkvmerge v82.0 ('I'm The President') 64-bit Error: no destination file name was given. $ env LC_CTYPE=en_US.UTF-8 mkvmerge terminate called after throwing an instance of 'std::runtime_error' what(): locale::facet::_S_create_c_locale name not valid Abort (core dumped) >This worked fine in 2022.4.0. Yeh the above code snippet is from their reworked locale handling in newer mkvtoolnix: https://gitlab.com/mbunkus/mkvtoolnix/-/commit/5cc251e27900dff888565edd680775a1d3502d5c Cheers, -- Jonathan Perkin - mnx.io - pkgsrc.smartos.org Open Source Complete Cloud www.tritondatacenter.com
I replied this:
Message-ID: <205ace48-b350-38cf-850c-fc9931b68dc7@jcea.es> Date: Tue, 12 Mar 2024 16:51:25 +0100 Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error To: smartos-discuss@lists.smartos.org References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> From: Jesus Cea <jcea@jcea.es> In-Reply-To: <ZfBzzJyHEb-Ujy64@mnx.io> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> X-getmail-retrieved-from-mailbox: INBOX On 12/3/24 16:25, Jonathan Perkin wrote: > I'll need to dig in further with the --enable-clocale option for > libstdc++ to see if we can improve this. Until then you might be able > to work around it by unsetting your locale environment variables, i.e.: Thanks for spending time on this. Much appreciated. Yes, if I do "LC_ACC=C", it works fine... but then I have issues with utf-8 file names. I can workaround it now with renames before and after, but it is... a pretty ugly kludge. A different issue: I am trying to compile "mkvtoolnix" in a 2023.4.0 pkgsrc zone (to test ideas & patches), and I get this error: """ --<root@PkgSrc>-(/data/chroot/dev-2023Q4-x86_64)-</data/pkgsrc/multimedia/mkvtoolnix>-- -> bmake clean ===> Cleaning for mkvtoolnix-81.0 --<root@PkgSrc>-(/data/chroot/dev-2023Q4-x86_64)-</data/pkgsrc/multimedia/mkvtoolnix>-- -> bmake package [...] /opt/local/bin/msgfmt: error while opening "/devices/pseudo/mm@0:null" for writing: No such file or directory *** Error code 1 Stop. bmake[1]: stopped in /data/pkgsrc/multimedia/mkvtoolnix *** Error code 1 Stop. bmake: stopped in /data/pkgsrc/multimedia/mkvtoolnix """ I wonder how are your doing the compilation... > https://gitlab.com/mbunkus/mkvtoolnix/-/commit/5cc251e27900dff888565edd680775a1d3502d5c Thanks for the pointer. Now I need to be able to compile it in pkgsrc zone O:-). Thanks for your time and expertise, Jonathan! Looking for progress on this.
Then I added the following message, triaging the root cause and proposing a temporal workaround:
Message-ID: <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es>
Date: Wed, 13 Mar 2024 02:04:34 +0100
Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with
locale error
To: smartos-discuss@lists.smartos.org
References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es>
<ZfBzzJyHEb-Ujy64@mnx.io>
From: Jesus Cea <jcea@jcea.es>
In-Reply-To: <ZfBzzJyHEb-Ujy64@mnx.io>
List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org>
On 12/3/24 16:25, Jonathan Perkin wrote:
> I've narrowed this down to a smaller test case:
>
> ---
> #include <codecvt>
> #include <iostream>
> #include <locale>
>
> int
> main()
> {
> auto const &locale_name = "en_US.UTF-8";
> auto utf8_locale = std::locale {
> std::locale{locale_name},
> new std::codecvt_utf8<wchar_t>
> };
> try {
> std::locale::global(utf8_locale);
> } catch (std::runtime_error &) {
> }
> }
This was useful, thanks. My C++ skills are null...
I have tried this:
Running your testcase with "truss -u libstdc++ ./a.out" and paying
attention to the abort message, I see that the failing function is
"libstdc++:_ZNSt6locale5facet18_S_create_c_localeERPiPKcS1_". Demangling
the C++ symbol (by hand!) I get
"std::locale::facet::_S_create_c_locale(int*&, char const*, int*)".
Looking around I found this link useful too:
https://stackoverflow.com/a/20181564 . The point is that GCC C++ locale
implementation is quite incomplete in other platforms beside Linux.
So I downloaded GCC 13.2.0 and checked the source code. The supported
implementations are in "gcc-13.2.0/libstdc++-v3/config/local/". Currently:
darwin/
dragonfly/
generic/
gnu/
ieee_1003.1-2001/
newlib/
vxworks/
No Solaris, so I guess we are hitting the "generic" path. The
implementation is quite appalling ("generic/c_locale.cc"):
"""
void
locale::facet::_S_create_c_locale(__c_locale& __cloc, const char* __s,
__c_locale)
{
// Currently, the generic model only supports the "C" locale.
// See http://gcc.gnu.org/ml/libstdc++/2003-02/msg00345.html
__cloc = 0;
if (strcmp(__s, "C"))
__throw_runtime_error(__N("locale::facet::_S_create_c_locale "
"name not valid"));
}
"""
The only supported option is "C" locale!!!!!. All others fails not
trying at all.
Studying other implementations, I found that "dragonfly" seems to
implemente complete functionality using "newlocale()". Illumos has
"newlocale()" too. Uhmmm...
The right way would be to try to compile "libstdc++" under Illumos using
dragonfly implementation or doing a new "Solaris" implementation that
actually works, but I don't understand how pkgsrc "lang/gcc13-libs"
works. A "bmake package" build a "gcc13-libs-13.2.0nb1.tgz" but the
process is different from other pkgsrc packages and nothing is compiled.
I guess the binary is downloaded from somewhere.This path is dead end to me.
I tried another path.
I wrote this code (the right way of doing it would be C++, but I don't
know C++, so I did it the hard way).
This is C:
"""
#include <stdio.h>
#include <locale.h>
void _ZNSt6locale5facet18_S_create_c_localeERPiPKcS1_(locale_t *x, const
char *s) {
*x = newlocale(LC_ALL_MASK, s, (locale_t)0);
printf("JCEA LD_PRELOAD Locale: '%s': %p!\n", s, *x);
}
"""
Notice I use the mangled function name, because this is C, not C++.
I compile this code with:
"""
gcc --shared -fpic jcea_preload_locale.c -o jcea_preload_locale.so
"""
I run "mkvmerge" as:
"""
LD_PRELOAD=/tmp/jcea_preload_locale.so mkvmerge -o áć.mkv
/home/transcoding/Process/XX.mkv
"""
Notice the unicode filename. I hope your email client can display it.
Execution works!!:
"""
[jcea@TEST /tmp]$ LD_PRELOAD=/tmp/jcea_preload_locale.so mkvmerge -o
áć.mkv /home/transcoding/Process/XX.mkv
JCEA LD_PRELOAD Locale: 'C': fffffc7feeda1800!
JCEA LD_PRELOAD Locale:
'LC_CTYPE=en_US.UTF-8;LC_NUMERIC=es_ES.UTF-8;LC_TIME=es_ES.UTF-8;LC_COLLATE=es_ES.UTF-8;LC_MONETARY=es_ES.UTF-8;LC_MESSAGES=en_US.UTF-8': 0!
[...]
The file 'áć.mkv' has been opened for writing.
The cue entries (the index) are being written...
Multiplexing took 0 seconds.
[jcea@TEST /tmp]$ ls -la á*
-rw-r--r-- 1 jcea jcea 270999 mar. 13 01:43 áć.mkv
"""
It is interesting that first call works (the pointer) but the funny
second locale fails (the 0 is a NULL, an error), but this works anyway.
Good enough for me, for now.
This code is a proof of concept only. It creates memory leaks in long
running processes with locale changing, and it doesn't implement many
other functions in the original source code, but it is good enough for
me, for now, and it is a proof of where is the root issue: libstdc++ in
GCC, locales in Illumos are not supported at all, although support seems
fairly trivial, in my opinion.
But I don't know C++ and GCC developers will pay ZERO attention to me,
for sure. I am nobody from nowhere and it is obvious that Illumos is not
a Tier 1 platform for them.
Anybody else can take care of this for upstream patching? Jonathan?
Thanks!
PS: I guess this could be a (big) local patch in pkgsrc if GCC guys
don't care, but I don't understand how pkgsrc "lang/gcc13-libs" works. I
see a package generated, but no compilation at all!.
Jason King added an interesting and useful side node:
From: Jason King <jason.brian.king@gmail.com> To: smartos-discuss <smartos-discuss@lists.smartos.org> Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error Date: Wed, 13 Mar 2024 01:10:04 +0000 Message-ID: <SJ0PR12MB673474F3C914C0E975E01BA9A42A2@SJ0PR12MB6734.namprd12.prod.outlook.com> References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> In-Reply-To: <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> Side note: to save yourself the pain of demangling by hand, you can use the demangle command: $ demangle '_ZNSt6locale5facet18_S_create_c_localeERPiPKcS1_' std::locale::facet::_S_create_c_locale(int*&, char const*, int*)
I will not paste more of that side thread. You can find it in the original thread.
Going back to the main thread, Jonathan Perkin added:
Date: Wed, 13 Mar 2024 08:19:45 +0000 From: Jonathan Perkin <jperkin@mnx.io> To: smartos-discuss <smartos-discuss@lists.smartos.org> Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error Message-ID: <ZfFhoZQ160FCdZcR@mnx.io> References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> In-Reply-To: <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> Just a couple of quick notes: * On 2024-03-13 at 01:06 GMT, Jesus Cea wrote: >I have tried this: Thanks for the input! I was looking through the same ideas yesterday, and was first trying to see if switching to the ieee_1003.1-2001 backend would help, but that doesn't even compile. I'll continue looking at this today. >The right way would be to try to compile "libstdc++" under Illumos >using dragonfly implementation or doing a new "Solaris" implementation >that actually works, but I don't understand how pkgsrc >"lang/gcc13-libs" works. A "bmake package" build a >"gcc13-libs-13.2.0nb1.tgz" but the process is different from other >pkgsrc packages and nothing is compiled. I guess the binary is >downloaded from somewhere.This path is dead end to me. The main GCC 13 package is at extra/gcc13: https://github.com/TritonDataCenter/pkgsrc-extra/tree/main/gcc13 I use my custom extra/gcc13 package instead of lang/gcc13 because we use the illumos patches, and it's easier to have my own package provide that, as well as simplifying the package significantly. As you've found out the -libs packages do not build anything, they simply repackage the libraries that are built in the main GCC packages. The reason for the separate -libs packages is so that all packages that require GCC runtime libraries (in reality most packages) can just depend on the smaller gcc13-libs package, instead of having to depend on the full gcc13 compiler suite. This makes a big difference: -rw-r--r-- 1 root root 267142642 Jan 13 12:04 gcc13-13.2.0.tgz -rw-r--r-- 1 root root 7456908 Jan 13 12:07 gcc13-libs-13.2.0.tgz Hope that helps,
Jonathan Perkin keeps digging:
Date: Wed, 13 Mar 2024 12:25:02 +0000
From: Jonathan Perkin <jperkin@mnx.io>
To: smartos-discuss <smartos-discuss@lists.smartos.org>
Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with
locale error
Message-ID: <ZfGbHu1ldUOiuC7M@mnx.io>
References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es>
<ZfBzzJyHEb-Ujy64@mnx.io>
<981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es>
<ZfFhoZQ160FCdZcR@mnx.io>
In-Reply-To: <ZfFhoZQ160FCdZcR@mnx.io>
List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org>
Ok so I started hacking on the DragonFly backend and now have it
building. It seems to work at least with minimal testing:
---
$ env \
LD_LIBRARY_PATH=/home/pbulk/build/extra/gcc13/work/build/x86_64-sun-solaris2.11/libstdc++-v3/src/.libs/ \
LANG=es_ES.UTF-8 \
mkvmerge
mkvmerge v82.0 ('I'm The President') 64-bit
Error: no fue asignado ningún nombre al archivo de salida.
mkvmerge -o salida [opciones generales] [opciones1] <archivo1> [@opciones-archivo.json] ...
...
Por favor, lea la página principal o la documentación HTML para
mkvmerge. En ella, se explican ciertos aspectos de una manera
más detallada que no son tan obvias desde este listado.
---
I had to stub out some of the code as we are missing some of the _l()
extensions (e.g. localeconv_l()), but this is mostly around LC_MONETARY
support so that will only support "C" for now. It's probably possible
to work around this by using setlocale() and localeconv() instead, but
I'm not yet confident that won't have side-effects doing so within the
guts of the C++ library.
I'll do some more testing and then look to merge this into trunk for
wider reach.
The fix seems good enough:
Date: Sat, 16 Mar 2024 08:28:03 +0000 From: Jonathan Perkin <jperkin@mnx.io> To: smartos-discuss <smartos-discuss@lists.smartos.org> Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error Message-ID: <ZfVYE3CFfGdNXGUK@mnx.io> References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> <ZfFhoZQ160FCdZcR@mnx.io> <ZfGbHu1ldUOiuC7M@mnx.io> In-Reply-To: <ZfGbHu1ldUOiuC7M@mnx.io> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> All the trunk sets have now been rebuilt with this change, and I've also switched the daily bulk builds to use the same compiler. No regressions detected so far. Cheers,
New releases will be fixed. Good. But what about current release?:
Message-ID: <969c687d-09a0-acdb-a6c3-0d96e42c98f5@jcea.es> Date: Thu, 21 Mar 2024 01:43:34 +0100 Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error To: smartos-discuss@lists.smartos.org References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> <ZfFhoZQ160FCdZcR@mnx.io> <ZfGbHu1ldUOiuC7M@mnx.io> <ZfVYE3CFfGdNXGUK@mnx.io> From: Jesus Cea <jcea@jcea.es> In-Reply-To: <ZfVYE3CFfGdNXGUK@mnx.io> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> On 16/3/24 9:28, Jonathan Perkin wrote: > All the trunk sets have now been rebuilt with this change, and I've also > switched the daily bulk builds to use the same compiler. No regressions > detected so far. Will this be backported to 23.4.0 LTS images?
There is hope, but not a promise:
Date: Thu, 21 Mar 2024 08:39:40 +0000 From: Jonathan Perkin <jperkin@mnx.io> To: smartos-discuss <smartos-discuss@lists.smartos.org> Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error Message-ID: <ZfvyTHphGmribdvo@mnx.io> References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> <ZfFhoZQ160FCdZcR@mnx.io> <ZfGbHu1ldUOiuC7M@mnx.io> <ZfVYE3CFfGdNXGUK@mnx.io> <969c687d-09a0-acdb-a6c3-0d96e42c98f5@jcea.es> In-Reply-To: <969c687d-09a0-acdb-a6c3-0d96e42c98f5@jcea.es> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> * On 2024-03-21 at 00:45 GMT, Jesus Cea wrote: >On 16/3/24 9:28, Jonathan Perkin wrote: >>All the trunk sets have now been rebuilt with this change, and I've >>also switched the daily bulk builds to use the same compiler. No >>regressions detected so far. > >Will this be backported to 23.4.0 LTS images? I'm still undecided. Ordinarily there's no way I'd consider backporting such a fundamental compiler change to a stable release. However, it clearly fixes a problem, and as yet I haven't been able to find any regressions (and due to the nature of the fix I wouldn't expect there to be any). I guess my question to you is how tied are you to running 23.4.0 compared to trunk? If it's quite inconvenient to use trunk for this then I'd be happy to consider backporting.
I replied this:
Message-ID: <dc6b55e8-a2fe-1c37-ca99-fd56e178369c@jcea.es> Date: Thu, 21 Mar 2024 15:31:13 +0100 Subject: Re: [smartos-discuss] Stock "mkvtoolnix" crashes on 2023.4.0 with locale error To: smartos-discuss@lists.smartos.org References: <15dec164-940e-17d7-a7db-dcdf19f9ec2d@jcea.es> <ZfBzzJyHEb-Ujy64@mnx.io> <981bd8a0-6b55-4095-6e1b-fc6482e86836@jcea.es> <ZfFhoZQ160FCdZcR@mnx.io> <ZfGbHu1ldUOiuC7M@mnx.io> <ZfVYE3CFfGdNXGUK@mnx.io> <969c687d-09a0-acdb-a6c3-0d96e42c98f5@jcea.es> <ZfvyTHphGmribdvo@mnx.io> From: Jesus Cea <jcea@jcea.es> In-Reply-To: <ZfvyTHphGmribdvo@mnx.io> List-Id: "smartos-discuss" <smartos-discuss.lists.smartos.org> On 21/3/24 9:39, Jonathan Perkin wrote: > I guess my question to you is how tied are you to running 23.4.0 > compared to trunk? If it's quite inconvenient to use trunk for this > then I'd be happy to consider backporting. This is a production zone, so I rather prefer to stay at 23.4.x. I have my LD_PRELOAD trick working here, so this is not catastrophic for me personally, but a LTS release with broken binaries is such a fundamental way...