[OpenSSL five]: HMAC algorithm analysis

1. Write in front

  Recently, due to work needs, I deeply and systematically studied the implementation of HMAC in OpenSSL. In order to lay a solid foundation for HMAC and help latecomers, I recorded some of my own debugging experience here.
  The code version of OpenSSL analyzed in this article is: openssl-1.1.1h
  path of hamc: crypto/hmac, which mainly includes 3 files hmac.c \ hm_pmeth.c \ hmeth.c.
  For a detailed analysis of the HMAC principle, see: Encryption Algorithm 2 HMAC

2. Main structure

• typedef struct hmac_ctx_st HMAC_CTX;
• typedef struct evp_md_st EVP_MD;
• typedef struct evp_md_ctx_st EVP_MD_CTX;
• typedef struct evp_pkey_ctx_st EVP_PKEY_CTX;
• typedef struct evp_pkey_st EVP_PKEY
• typedef struct evp_pkey_method_st EVP_PKEY_METHOD;

2.1 HAMC_CTX (file: ossl_typ.h >>> evp_local.h)

  This structure belongs to a custom structure of the OpenSSL soft algorithm. If the OpenSSL soft algorithm is used, this structure will be used, but if the calling engine (Engine) hardware implements HMAC, this structure will generally be used.

struct hmac_ctx_st {
    
    
    const EVP_MD *md;    /* 摘要算法的结构体，每种算法都有这么一个结构体，类似算法的句柄 */
    EVP_MD_CTX *md_ctx;  /* 摘要算法的上下文 */
    EVP_MD_CTX *i_ctx;	 /* i代表ipad（内部秘钥），是ipad散列运算的上下文 */
    EVP_MD_CTX *o_ctx;	 /* o代表opad（外部秘钥），是opad散列运算的上下文 */
};

typedef struct hmac_ctx_st HMAC_CTX;

2.2 EVP_MD (file: ossl_typ.h >>> evp.h)

  The structure of the digest algorithm, similar to the handle of the digest algorithm. This structure defines a collection of abstract methods for general summary calculation, which can be understood as a subclass of EVP_MD_CTX.

struct evp_md_st {
    
    
    int type;
    int pkey_type;
    int md_size; 		/* digest的长度（这个是与算法有关的，比如sha256，摘要值的长度为32字节） */
    unsigned long flags;
    int (*init) (EVP_MD_CTX *ctx);	/* 初始化函数 */
    int (*update) (EVP_MD_CTX *ctx, const void *data, size_t count);	/* 中间过程运算，更新函数 */
    int (*final) (EVP_MD_CTX *ctx, unsigned char *md);	/* 最后一笔运算，用于获取摘要值，不在进行数据的摘要运算 */
    int (*copy) (EVP_MD_CTX *to, const EVP_MD_CTX *from);	/* 复制函数 */
    int (*cleanup) (EVP_MD_CTX *ctx);	/* 复位函数 */	
    int block_size;	/* md的块大小 */
    int ctx_size; 	/* how big does the ctx->md_data need to be */
    /* control function */
    int (*md_ctrl) (EVP_MD_CTX *ctx, int cmd, int p1, void *p2);
} /* EVP_MD */ ;

typedef struct evp_md_st EVP_MD;

2.3 EVP_MD_CTX (file: ossl_typ.h >>> evp_loacl.h)

   The context of the digest algorithm.
   Since it is context, it must contain "summary" and "data". *md_data is the data pointer of the abstract, and the space generally needs to be applied for by yourself.
   For this article, the variable in the structure is " *pctx", which points to the context of pkey (hmac is divided into pkey class in openssl), and the definition of EVP_PKEY_CTX is shown in 2.4.

struct evp_md_ctx_st {
    
    
    const EVP_MD *digest;		/* 摘要 */
    ENGINE *engine;             /* functional reference if 'digest' is ENGINE-provided */
    unsigned long flags;		
    void *md_data;	/* 指向摘要的具体上下文，这个一般有用户自己定义（在openssl的软算法中指向HMAC_CTX所声明的结构体） */
    /* Public key context for sign/verify */
    EVP_PKEY_CTX *pctx; /* 签名（auth）的上下文，openssl将auth归为了pkey类，但是它的运算过程与md运算的过程类似 */
    /* Update function: usually copied from EVP_MD */
    int (*update) (EVP_MD_CTX *ctx, const void *data, size_t count);
} /* EVP_MD_CTX */ ;

typedef struct evp_md_ctx_st EVP_MD_CTX;

2.4 EVP_PKEY_CTX (file: ossl_typ.h >>> evp.h)

  The context structure of pkey is as follows:

struct evp_pkey_ctx_st {
    
    
    /* Method associated with this operation */
    const EVP_PKEY_METHOD *pmeth;
    /* Engine that implements this method or NULL if builtin */
    ENGINE *engine;
    /* Key: may be NULL */
    EVP_PKEY *pkey;
    /* Peer key for key agreement, may be NULL */
    EVP_PKEY *peerkey;
    /* Actual operation */
    int operation;
    /* Algorithm specific data */
    void *data;
    /* Application specific data */
    void *app_data;
    /* Keygen callback */
    EVP_PKEY_gen_cb *pkey_gencb;
    /* implementation specific keygen data */
    int *keygen_info;
    int keygen_info_count;
} /* EVP_PKEY_CTX */ ;

typedef struct evp_pkey_ctx_st EVP_PKEY_CTX;

2.5 EVP_PKEY (file: ossl_typ.h >>> evp.h)

  The structure of the pkey algorithm, similar to the handle of the pkey algorithm.

/*
 * Type needs to be a bit field Sub-type needs to be for variations on the
 * method, as in, can it do arbitrary encryption....
 */
struct evp_pkey_st {
    
    
    int type;
    int save_type;
    CRYPTO_REF_COUNT references;
    const EVP_PKEY_ASN1_METHOD *ameth;
    ENGINE *engine;
    ENGINE *pmeth_engine; /* If not NULL public key ENGINE to use */
    union {
    
    
        void *ptr;
# ifndef OPENSSL_NO_RSA
        struct rsa_st *rsa;     /* RSA */
# endif
# ifndef OPENSSL_NO_DSA
        struct dsa_st *dsa;     /* DSA */
# endif
# ifndef OPENSSL_NO_DH
        struct dh_st *dh;       /* DH */
# endif
# ifndef OPENSSL_NO_EC
        struct ec_key_st *ec;   /* ECC */
        ECX_KEY *ecx;           /* X25519, X448, Ed25519, Ed448 */
# endif
    } pkey;
    int save_parameters;
    STACK_OF(X509_ATTRIBUTE) *attributes; /* [ 0 ] */
    CRYPTO_RWLOCK *lock;
} /* EVP_PKEY */ ;

typedef struct evp_pkey_st EVP_PKEY;

2.6 EVP_PKEY_METHOD (file: ossl_typ.h >>> evp.h)

  This structure defines a collection of general mac computing abstract methods.

struct evp_pkey_method_st {
    
    
    int pkey_id;
    int flags;
    int (*init) (EVP_PKEY_CTX *ctx);
    int (*copy) (EVP_PKEY_CTX *dst, EVP_PKEY_CTX *src);
    void (*cleanup) (EVP_PKEY_CTX *ctx);
    int (*paramgen_init) (EVP_PKEY_CTX *ctx);
    int (*paramgen) (EVP_PKEY_CTX *ctx, EVP_PKEY *pkey);
    int (*keygen_init) (EVP_PKEY_CTX *ctx);
    int (*keygen) (EVP_PKEY_CTX *ctx, EVP_PKEY *pkey);
    int (*sign_init) (EVP_PKEY_CTX *ctx);
    int (*sign) (EVP_PKEY_CTX *ctx, unsigned char *sig, size_t *siglen,
                 const unsigned char *tbs, size_t tbslen);
    int (*verify_init) (EVP_PKEY_CTX *ctx);
    int (*verify) (EVP_PKEY_CTX *ctx,
                   const unsigned char *sig, size_t siglen,
                   const unsigned char *tbs, size_t tbslen);
    int (*verify_recover_init) (EVP_PKEY_CTX *ctx);
    int (*verify_recover) (EVP_PKEY_CTX *ctx,
                           unsigned char *rout, size_t *routlen,
                           const unsigned char *sig, size_t siglen);
    int (*signctx_init) (EVP_PKEY_CTX *ctx, EVP_MD_CTX *mctx);
    int (*signctx) (EVP_PKEY_CTX *ctx, unsigned char *sig, size_t *siglen,
                    EVP_MD_CTX *mctx);
    int (*verifyctx_init) (EVP_PKEY_CTX *ctx, EVP_MD_CTX *mctx);
    int (*verifyctx) (EVP_PKEY_CTX *ctx, const unsigned char *sig, int siglen,
                      EVP_MD_CTX *mctx);
    int (*encrypt_init) (EVP_PKEY_CTX *ctx);
    int (*encrypt) (EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen,
                    const unsigned char *in, size_t inlen);
    int (*decrypt_init) (EVP_PKEY_CTX *ctx);
    int (*decrypt) (EVP_PKEY_CTX *ctx, unsigned char *out, size_t *outlen,
                    const unsigned char *in, size_t inlen);
    int (*derive_init) (EVP_PKEY_CTX *ctx);
    int (*derive) (EVP_PKEY_CTX *ctx, unsigned char *key, size_t *keylen);
    int (*ctrl) (EVP_PKEY_CTX *ctx, int type, int p1, void *p2);
    int (*ctrl_str) (EVP_PKEY_CTX *ctx, const char *type, const char *value);
    int (*digestsign) (EVP_MD_CTX *ctx, unsigned char *sig, size_t *siglen,
                       const unsigned char *tbs, size_t tbslen);
    int (*digestverify) (EVP_MD_CTX *ctx, const unsigned char *sig,
                         size_t siglen, const unsigned char *tbs,
                         size_t tbslen);
    int (*check) (EVP_PKEY *pkey);
    int (*public_check) (EVP_PKEY *pkey);
    int (*param_check) (EVP_PKEY *pkey);

    int (*digest_custom) (EVP_PKEY_CTX *ctx, EVP_MD_CTX *mctx);
} /* EVP_PKEY_METHOD */ ;

typedef struct evp_pkey_method_st EVP_PKEY_METHOD;

3. Main function

  Due to the needs of the work, this time only the related functions of hm_pmeth.c and hamc.c are studied, and the functions in hm_pmeth.c and hmac.c are analyzed one by one.
  In 1.1.1, most data structures are no longer available to users, which makes more sense from an encapsulation point of view. If you can't find the structure definition in the header file, you might as well search the source code.

3.1 Main functions in hmac.c

• HMAC_CTX HMAC_CTX_new(void)
  (1) Create a HAMC_CTX context structure (that is, allocate a memory space for the context structure).
• int HMAC_Init_ex(HMAC_CTX *ctx, const void *key, int len, const EVP_MD *md, ENGINE *impl)
  (1) Initialize the HAMC_CTX context structure, key is the secret key, len is the length of the secret key, and md is the set of functions for calculating hash (digest handle)
  (2) If the length of the key is greater than the "block size", you need to perform a hash operation on the key first. If the length of the key is less than the "block size", fill it with "0" until the length of the key equal to "block size".
  (3) Calculate the ipad, and calculate the hash value of the ipad, and store it in the i_ctx context;
  (4) Calculate the opad, and calculate the hash value of the opad, and store it in the o_ctx context;
  (5) Copy ctx->i_ctx to ctx- In >md_ctx, the purpose of this move is that according to the definition of the hamc algorithm, the ipad first participates in the hash calculation.
• int HMAC_Init(HMAC_CTX *ctx, const void *key, int len, const EVP_MD *md)
  (1) This function is used to be compatible with projects developed according to earlier versions, and directly calls HMAC_Init_ex to implement.
• int HMAC_Update(HMAC_CTX *ctx, const unsigned char *data, size_t len)
  (1) Call EVP_DigestUpdate to implement hash operation (it is fully seen that there are many similarities between calculating mac and calculating hash).
• int HMAC_Final(HMAC_CTX *ctx, unsigned char *md, unsigned int *len)
  (1) Call the function EVP_DigestFinal_ex() to get the hash value of the update operation and put it in buf;
  (2) Call the function EVP_MD_CTX_copy_ex() to copy the i_ctx context to In md_ctx;
  (3) Call the function EVP_DigestUpdate() to calculate the hash value;
  (4) Call the function EVP_DigestFinal_ex() again to obtain the final hash (digest) value.
  Through the above 4 steps of operation, according to the requirements of the algorithm, the opadkey is spliced ​​to the front of buf to realize the final result of hash calculation.
• void HMAC_CTX_free(HMAC_CTX *ctx)
  (1) Release the HAMC_CTX context structure (here, pay special attention to the need to release layer by layer, release the innermost layer first, and then release the outermost layer).
• unsigned char *HMAC(const EVP_MD *evp_md, const void *key, int key_len, const unsigned char *d, size_t n, unsigned char *md, unsigned int *md_len) (1) This function realizes the calculation of a single hash value
  , is a combination of the above functions.

3.2 Main functions in hm_pmeth.c

  There are many pkey operation functions defined in the structure EVP_PKEY_METHOD, but most of them may not be used. The following functions are mainly implemented in hm_pmeth.c. In the actual application development (engine development), we also rely on the gourd Draw a scoop to realize these functions.
  Key structure:

/* HMAC pkey context structure */
typedef struct {
    
    
    const EVP_MD *md;           /* MD for HMAC use */
    ASN1_OCTET_STRING ktmp;     /* Temp storage for key */
    HMAC_CTX *ctx;
} HMAC_PKEY_CTX;

• static int pkey_hmac_init(EVP_PKEY_CTX *ctx)
  (1) Initialize the HMAC_PKEY_CTX context structure and assign it to ctx->data;
  (2) The essential function of this function is to allocate space for the data variable of ctx.
• static int pkey_hmac_copy(EVP_PKEY_CTX *dst, EVP_PKEY_CTX *src)
  (1) Assignment context.
• static void pkey_hmac_cleanup(EVP_PKEY_CTX *ctx)
  (1) Clear and reset.
• static int pkey_hmac_keygen(EVP_PKEY_CTX *ctx, EVP_PKEY *pkey)
  (1) The important realization of this function is to copy the secret key in ctx->data to pkey. This function implements the transfer of the secret key instead of regenerating it .
• static int hmac_signctx_init(EVP_PKEY_CTX *ctx, EVP_MD_CTX *mctx)
  (1) This function is mainly to specify the update function for digest operation for mctx context.
• static int hmac_signctx(EVP_PKEY_CTX *ctx, unsigned char *sig, size_t *siglen, EVP_MD_CTX *mctx)
  (1) If mctx is empty, only the length of the summary value is returned.
  (2) If mctx is not empty, call HMAC_Final() to get the final summary value.
• static int pkey_hmac_ctrl(EVP_PKEY_CTX *ctx, int type, int p1, void *p2)
  (1) EVP_PKEY_CTRL_SET_MAC_KEY: Write the key into ctx->data, so that the key is saved in the ctx context.
  (2) EVP_PKEY_CTRL_MD: Associate the related md operation for the hamc operation (because the hamc operation is essentially a digest operation, the function set of the digest must be specified).
  (3) EVP_PKEY_CTRL_DIGESTINIT: Get the key from ctx->pkey->pkey.ptr, and perform the initialization operation of hamc. (If you switch to engine operation here, the key will be saved in a custom context for hardware calls, no software maintenance is required, openssl is software that maintains the key)
• static int pkey_hmac_ctrl_str(EVP_PKEY_CTX *ctx, const char *type, const char *value)

4. Software implementation

  After analyzing the above functions, we implemented the engine driver in the same mode and wrote the sample code. The following focuses on analyzing the sample code to sort out the execution process of the code.
  First, post the code I have written and verified, as follows:

/* 明文 */
static const unsigned char P[] = {
    
      
    0x1A, 0x1E, 0x1F, 0x2F, 0x3F, 0x4F, 0xFA, 0xBD, 0xED, 0xCD, 0xFA, 0xFC, 0xCA, 0xDA, 0xDB, 0x12,
    0x34, 0x56, 0x78, 0x90, 0x9A, 0x1D, 0x11, 0x1E, 0x12, 0x6C, 0x36, 0xDD, 0xFF, 0x12, 0x9A, 0x0F,
    0x22, 0x33, 0x44, 0x55, 0x66, 0x77, 0x88, 0x99, 0x00, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0xFF,
    0xDD, 0x12, 0x33, 0x44, 0x01, 0x12, 0x4A, 0x3F, 0x1A, 0x2B, 0xC8, 0x59, 0x6A, 0x05, 0x85, 0xE0,
};
/* 秘钥 */
static const unsigned char K[] = {
    
    
    0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44,
    0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44,
    0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44,
    0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44,
    0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44,
    0x11, 0x11, 0x22, 0x22, 0x33, 0x33, 0x44, 0x44,
};
/* 通过sha1计算的hamc值 */
static const unsigned char E_hamc_sha1[] = {
    
    
    0xFC, 0xE9, 0xFD, 0xB7, 0x95, 0x75, 0x3B, 0xFA, 0x5D, 0xC1, 0xF5, 0x8B, 0x4B, 0x25, 0x17, 0x33, 
    0xE5, 0x29, 0xD4, 0x04,
};
/* 自定义的结构体 */
struct test_sign {
    
    
    const char *name;
    unsigned int nid;
    const char *algname;
    const unsigned char *plaintext;
    const unsigned char *key;
    const unsigned char *mac;
    int psize;
    int keylen;
};

static struct test_sign test_signs[] = {
    
    
    {
    
    
        .name = "HMAC(md5)",
        .nid = EVP_PKEY_HMAC,
        .algname = "MD5",
        .plaintext = P,
        .key = K,
        .mac = E_hamc_md5,
        .psize = sizeof(P),
        .keylen = sizeof(K)
    },
    {
    
    0};
}

static int test_hmac(struct test_sign *t)
{
    
    
    int ret = SUCCESS, test;
    EVP_MD_CTX *mctx = NULL;
    EVP_PKEY_CTX *pctx = NULL, *genctx = NULL;
    EVP_PKEY *pkey = NULL;
    const EVP_MD *md = NULL;
    unsigned char mac[EVP_MAX_MD_SIZE];
    size_t mac_len = 0;
	
	/* key的生成过程 */
    genctx = EVP_PKEY_CTX_new_id(t->nid, NULL);	/* 通过nid获取 EVP_PKEY_CTX 上下文 */
    EVP_PKEY_keygen_init(genctx);	/* 对新申请的上下文进行初始化，主要用户内存的申请、数据的填充等 */
    EVP_PKEY_CTX_set_mac_key(genctx, t->key, t->keylen);	/* 将key设置到 genctx 上下文中 */
    EVP_PKEY_keygen(genctx, &pkey);	/* 将key复制到pkey中 */
    EVP_PKEY_CTX_free(genctx);	/* 释放 EVP_PKEY_CTX 上下文 */

	/* 通过sha1计算mac值 */
    md = EVP_get_digestbyname(t->algname);	/* 通过算法名称获取md */
    mctx = EVP_MD_CTX_new();	/* 创建一个全新的 EVP_MD_CTX 上下文*/
    EVP_DigestSignInit(mctx, &pctx, md, NULL, pkey);	/* 将md、pkey与mctx进行绑定 */
    EVP_DigestSignUpdate(mctx, t->plaintext, t->psize);	/* 计算摘要值 */
    EVP_DigestSignFinal(mctx, NULL, &mac_len);	/* 获取mac值的长度 */
    EVP_DigestSignFinal(mctx, mac, &mac_len);	/* 获取mac值 */

    /* check */
    TEST_ASSERT(((mac_len == sizeof(t->mac)) && (!memcmp(mac, t->mac, mac_len))),
                t->name, "digest");
    ret |= test;
	
	/* 释放内存 */
    EVP_PKEY_CTX_free(pctx);
    EVP_MD_CTX_free(mctx);
    EVP_PKEY_free(pkey);
    return ret;
}

  The calculation of the mac value is mainly divided into two steps: the first step is to generate the secret key, and the second step is to calculate the mac value (calculate the mac value by calculating the hash).

4.1 Key generation

• EVP_PKEY_CTX_new_id(): Get the context genctx of EVP_PKEY_CTX type through nid;
• EVP_PKEY_keygen_init(): Initialize the context of the new application, the application of the main user memory, the filling of data, etc. (if the engine is registered with openssl and the engine supports hamc operations, this function will call the init function of the engine, otherwise it will directly call openssl init function);
• EVP_PKEY_CTX_set_mac_key(): Set the key to the genctx context.
• EVP_PKEY_keygen(): /* copy key to pkey*/
• EVP_PKEY_CTX_free(): /* Release the genctx context, so far genctx has come to an end */
    The purpose of the above series of operations is to put the key in EVP_PKEY *pkey. This is the way I can achieve it at present. I don’t know if it is possible Put the key directly into pkey.

4.2 mac computing

• EVP_get_digestbyname(): Get EVP_MD md (a collection of functions for digest operations) by algorithm name;
• EVP_MD_CTX_new() : Create a digest context
• EVP_DigestSignInit(): This function mainly does the following things
  (1) If mctx->pctx is empty, the new window pctx context;
  (2) Specify the update function of the hash operation for mctx->update, and at the same time mctx-> pctx->operation = EVP_PKEY_OP_SIGNCTX;
  (3) Associate md with pctx;
  (4) Perform mctx initialization.
• EVP_DigestSignUpdate(): Calculates the digest value.
• EVP_DigestSignFinal(): Get mac length or mac value.

5. Summary

• In openssl, the calculation of hamc is classified as pkey calculation, but it has a lot of display with the calculation of digest. The main difference is that the calculation of digest requires a secret key, while the calculation of hamc requires a secret key, and the secret key is also There are two ipad_key and opad_key, and both of them can be generated by our secret key.
• The calculation of hamc has a process of key generation. It is not so much the key generation as the copying of the key. In fact, it is to put the key at the ptr position in the EVP_PKEY structure (memory needs to be applied for by yourself).
• EVP_MD_CTXThe context is included in the following , EVP_PKEY_CTXbecause essentially the hamc operation is also calculated with the set of function interfaces of digest.
• The hamc soft algorithm of openssl maintains ipad_key and opad_key by itself. In fact, when our hardware is implemented, it is maintained by the hardware, so the hardware implementation is slightly simpler than the software process.