Bug#582518: libtheora0: Using -funroll-loops seems to cause miscompilation

[Sjoerd Simons]

Package: libtheora0
Version: 1.1.1+dfsg.1-3
Severity: normal

Hey,

Long story so bare with me. I've been working on a gstreamer application, which
uses one or more theora encoders. For some reason i kept getting weird crashes
that i couldn't explain. With valgrind i kept hitting the following:

$ valgrind gst-launch-0.10 videotestsrc ! queue ! theoraenc ! fakesink

==29930== Memcheck, a memory error detector
==29930== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==29930== Using Valgrind-3.5.0-Debian and LibVEX; rerun with -h for copyright
info
==29930== Command: gst-launch-0.10 videotestsrc ! queue ! theoraenc ! fakesink
==29930== 
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
==29930== Thread 2:
==29930== Invalid write of size 2
==29930==    at 0x9180D10: oc_enc_tokenize_ac (tokenize.c:636)
==29930==    by 0x91632C0: oc_enc_block_transform_quantize (analyze.c:770)
==29930==    by 0x91664F7: oc_enc_mb_transform_quantize_luma (analyze.c:889)
==29930==    by 0x916C243: oc_enc_analyze_intra (analyze.c:1282)
==29930==    by 0x9177206: oc_enc_compress_keyframe (encode.c:1161)
==29930==    by 0x91774AD: th_encode_ycbcr_in (encode.c:1549)
==29930==    by 0x8B2FD00: theora_enc_chain (gsttheoraenc.c:1021)
==29930==    by 0x4E8368C: gst_pad_chain_data_unchecked (gstpad.c:4131)
==29930==    by 0x4E83F4D: gst_pad_push_data (gstpad.c:4360)
==29930==    by 0x890C6D8: gst_queue_loop (gstqueue.c:1083)
==29930==    by 0x4EABDBA: gst_task_func (gsttask.c:271)
==29930==    by 0x55B752E: g_thread_pool_thread_proxy (gthreadpool.c:315)
==29930==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==29930== 
Caught SIGSEGV accessing address (nil)
New clock: GstSystemClock
[1]    29930 killed     valgrind gst-launch-0.10 videotestsrc ! queue !
theoraenc ! fakesink

Not great, tracking down the code that line of tokenize.c just fills a stack
allocate array and should work fine. To be able to debug a bit better i
recompiled without optimisations and the bug magically went away...

After some trail and error it seems that the combination of -finline-functions
and -funroll-loops is the cause of the issue when using GCC 4.4, with GCC 4.3
everything works fine as well..

Figuring out what actually goes wrong in GCC here is probably going to take
quite some time, so i'd suggest compiling theora without -funroll-loops for
now.

  Sjoerd

-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (101, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.33-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages libtheora0 depends on:
ii  libc6                       2.10.2-9     Embedded GNU C Library: Shared lib
ii  libogg0                     1.2.0~dfsg-1 Ogg bitstream library

libtheora0 recommends no packages.

libtheora0 suggests no packages.

-- no debconf information