[Pkg-virtualbox-devel] Bug#828593: virtualbox: FTBFS with openssl 1.1.0

[Kurt Roeckx]

On Wed, Sep 07, 2016 at 11:38:01PM +0200, Sebastian Andrzej Siewior wrote:
> On 2016-06-26 12:24:40 [+0200], Kurt Roeckx wrote:
> > If you have problems making things work, feel free to contact us.
> 
> HALP!
> 
> It builds against old & new ssl and I am proud what I managed in
> rdssl_cert_to_rkey(). However they dereference
> EVP_MD->required_pkey_type which vanished in 1.1 and I have no idea what
> it was and its purpose was. They don't even assign anything to it in
> 1.0.2h.

I think this is the relevant commit:
commit 7f572e958b13041056f377a62d3219633cfb1e8a
Author: Dr. Stephen Henson <steve at openssl.org>
Date:   Wed Dec 2 13:57:04 2015 +0000

    Remove legacy sign/verify from EVP_MD.

    Remove sign/verify and required_pkey_type fields of EVP_MD: these are a
    legacy from when digests were linked to public key types. All signing is
    now handled by the corresponding EVP_PKEY_METHOD.

    Only allow supported digest types in RSA EVP_PKEY_METHOD: other algorithms
    already block unsupported types.

    Remove now obsolete EVP_dss1() and EVP_ecdsa().

    Reviewed-by: Richard Levitte <levitte at openssl.org>

The code seems to be doing: The signature algorithm was
sha1WithRSAEncryption, so the public key they passed us must
have been an RSA key. But from what I understand, what the code
really did was: it's an EVP_sha1(), it's an an RSA key. But if it
was EVP_sha256(), it was an unknown key type. What they do might
have worked in the past by accident. But it clearly doesn't work
properly.

I don't think there is a way of doing what they want to do.  They
should really just pass the type of the public key to the
function.


Kurt