[Pkg-db-devel] Bug#421946: OpenLDAP 2.3.38 and Berkeley DB 4.6.19 (fwd)

[Clint Adams]

On Tue, Aug 28, 2007 at 02:03:57PM -0700, Quanah Gibson-Mount wrote:
> Here's the relevant information.
>
> Looks like x86_64 requires a specific option.  Maybe it'd be best to 
> compile it that way for all platforms.

Are there reproducible tests/benchmarks that we can use to figure out
whether or not this actually makes sense on any given architecture?

> ------------ Forwarded Message ------------
> Date: Tuesday, August 28, 2007 1:47 PM -0700
> From: Howard Ch
> To: Quanah Gibson-Mount
> Cc: openldap-software at openldap.org
> Subject: Re: OpenLDAP 2.3.38 and Berkeley DB 4.6.19
>
> Quanah Gibson-Mount wrote:
>> --On Tuesday, August 28, 2007 2:44 PM -0400 matthew sporleder
>> <msporleder at gmail.com> wrote:
>>
>>> That's an unfortunate development from oracle considering the
>>> presentation that was released showing bdb4.6 to be very fast.
>>
>> Yup.  I believe Howard is working with Sleepycat to find out what was
>> done  to 4.6 to make it suddenly be 3-4 times slower instead of
>> significantly  faster as it was in the early releases.
>
> Yes, it's been a long conversation so far. While their new memory manager
> (inspired by Jong-Hyuk Choi's research on BerkeleyDB performance in
> OpenLDAP) is a great improvement, their new lock manager is not.
>
> This may not be an issue on all platforms. On x86 though, they have a
> hybrid mutex mechanism which is enabled by default. It uses both assembly
> language spinlocks and pthread mutexes, first spinning in the assembly
> language lock some number of times before falling back onto the mutex. They
> claimed that this improved performance in their tests, because pthread
> mutexes can be very expensive on some platforms. In my tests on x86_64
> Linux however, it just forced CPU usage to 100% (200% actually, dualcore)
> and slowed down overall throughput.
>
> You can give an explicit "--with-mutex=POSIX/pthread" argument when
> configuring BerkeleyDB to avoid the hybrid mutex scheme, in which case
> performance of BDB 4.6.19 seems to match what we obtained with BDB 4.6.3.
>
> If you were using BerkeleyDB 4.5.20 before, the early BDB 4.6 releases will
> work well for you. The only difference between 4.5.20 and 4.6.1 was in the
> memory manager. 4.6.2 just tweaked some portability issues to support BREW
> (Qualcomm's cellphone programming environment). 4.6.3 added the ability to
> specify different cache priorities per database operation. (I didn't track
> what changed between 4.6.3 and 4.6.18.)
>
> And for anyone curious - you can read ITS#3851
> http://www.openldap.org/its/index.cgi/Development?id=3851 for the
> background on the problems in BerkeleyDB's memory allocator. While there
> was a fair amount of debate as to whether Jong's proposed solution was of
> any benefit, it was pretty clear that the existing code in BerkeleyDB could
> be improved. But the changes in the memory allocator may not visibly affect
> you if you're running a small-to-medium sized database. It's only apparent
> when the total volume of data is much larger than the BerkeleyDB cache,
> because that's the condition that leads to memory fragmentation, and it's
> the fragmentation that causes the allocator's performance to degrade.
>
> The same kind of problem affects the slapd entry cache, when the number of
> active entries is much larger than the entry cache, and you're using the
> glibc malloc. (At least, for glibc 2.3 and older. I haven't retested with
> glibc 2.5 yet.)
>
> To sum up - if you're already using BDB 4.5.20 with OpenLDAP 2.3 and the
> performance is acceptable, I wouldn't be in any hurry to upgrade to BDB
> 4.6. Releases 4.6.18 and 4.6.19 are incompatible with OpenLDAP 2.3. BDB
> 4.6.1 is a drop-in replacement for 4.5.20 though.
> -- 
>   -- Howard Chu
>   Chief Architect, Symas Corp.  http://www.symas.com
>   Director, Highland Sun        http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP     http://www.openldap.org/project/
>
> ---------- End Forwarded Message ----------