1. 引言
Campanelli等人 2020年论文《Vector Commitment Techniques and Applications to Verifiable Decentralized Storage》对应的代码实现：
https://github.com/nicola/rust-yinyan
是在https://github.com/dignifiedquire/rust-accumulators （主要基于[BBF19]2018年论文《Batching Techniques for Accumulators with Applications to IOPs and Stateless Blockchains》——是基于strong RSA assumption in groups of unknown order来实现的，其基本原理为：
针对bit vector
，构建一个相应的co-prime【co-prime这个属性很好，可以保证某个数值不可能在两个以上的位置存在，否则与co-prime矛盾。】 vector
，
的个数为
。若
，则提供an inclusion proof of
；若
，则提供an exclusion proof of
。引申到更通用的情况是，任意的vector，其元素可由
组成，相应的open可引申为
个位置的batch opening。） 代码基础上添加了src/vc/yinyan.rs等文件来做了相应的算法验证。（具体可参见博客 Vector Commitments代码实现 2.1节内容。）
https://github.com/nicola/rust-yinyan 主要对论文《Vector Commitment Techniques and Applications to Verifiable Decentralized Storage》中的第一种SVC算法进行了实现，参见博客 Vector Commitment Techniques and Applications to Verifiable Decentralized Storage学习笔记 中6.1节内容。
不过https://github.com/nicola/rust-yinyan 代码实现与实际论文有差异，并且有点小bug，可参看 https://github.com/3for/rust-yinyan 中代码实现已修复相应的bug，保证了cargo test，cargo run等功能正常。
单核4G内存Ubuntu16.04虚拟机内cargo bench表现为：
Warning: Sample sizes < 10 will be disallowed in Criterion.rs 0.3.0.
Gnuplot not found, disabling plotting
NON_PRE_bench_bbf_commit_VECSZ=512_OPNSZ=64
time: [74.114 ms 75.257 ms 89.801 ms]
NON_PRE_bench_yinyan_commit_VECSZ=512_OPNSZ=64
time: [1.3965 s 2.1971 s 2.4840 s]
NON_PRE_bench_bbf_batch_open_VECSZ=512_OPNSZ=64
time: [168.90 ms 171.17 ms 197.71 ms]
NON_PRE_bench_yinyan_batch_open_VECSZ=512_OPNSZ=64
time: [142.56 ms 144.83 ms 168.00 ms]
NON_PRE_bench_bbf_verify_VECSZ=512_OPNSZ=64
time: [6.4363 ms 6.5631 ms 8.0323 ms]
NON_PRE_bench_yy_verify_VECSZ=512_OPNSZ=64
time: [2.8980 ms 2.9556 ms 3.4921 ms]
NON_PRE_bench_bbf_commit_VECSZ=1024_OPNSZ=128
time: [155.96 ms 158.65 ms 179.40 ms]
NON_PRE_bench_yinyan_commit_VECSZ=1024_OPNSZ=128
time: [2.7085 s 3.4412 s 3.7044 s]
NON_PRE_bench_bbf_batch_open_VECSZ=1024_OPNSZ=128
time: [350.06 ms 354.62 ms 394.46 ms]
NON_PRE_bench_yinyan_batch_open_VECSZ=1024_OPNSZ=128
time: [281.37 ms 284.76 ms 327.09 ms]
NON_PRE_bench_bbf_verify_VECSZ=1024_OPNSZ=128
time: [6.6859 ms 6.7731 ms 7.8816 ms]
NON_PRE_bench_yy_verify_VECSZ=1024_OPNSZ=128
time: [2.0716 ms 2.0978 ms 2.4307 ms]
NON_PRE_bench_bbf_commit_VECSZ=2048_OPNSZ=256
time: [365.31 ms 459.23 ms 501.64 ms]
NON_PRE_bench_yinyan_commit_VECSZ=2048_OPNSZ=256
time: [3.3679 s 5.0147 s 5.6279 s]
NON_PRE_bench_bbf_batch_open_VECSZ=2048_OPNSZ=256
time: [789.64 ms 798.51 ms 872.60 ms]
NON_PRE_bench_yinyan_batch_open_VECSZ=2048_OPNSZ=256
time: [601.52 ms 606.58 ms 672.21 ms]
NON_PRE_bench_bbf_verify_VECSZ=2048_OPNSZ=256
time: [8.1450 ms 8.3215 ms 10.254 ms]
NON_PRE_bench_yy_verify_VECSZ=2048_OPNSZ=256
time: [3.3099 ms 3.3725 ms 4.0539 ms]
NON_PRE_bench_bbf_commit_VECSZ=4096_OPNSZ=512
time: [698.47 ms 709.04 ms 802.36 ms]
NON_PRE_bench_yinyan_commit_VECSZ=4096_OPNSZ=512
time: [6.5220 s 10.038 s 11.239 s]
NON_PRE_bench_bbf_batch_open_VECSZ=4096_OPNSZ=512
time: [1.5950 s 1.6223 s 1.8070 s]
NON_PRE_bench_yinyan_batch_open_VECSZ=4096_OPNSZ=512
time: [1.2118 s 1.2219 s 1.3454 s]
NON_PRE_bench_bbf_verify_VECSZ=4096_OPNSZ=512
time: [8.4131 ms 8.5793 ms 10.349 ms]
NON_PRE_bench_yy_verify_VECSZ=4096_OPNSZ=512
time: [3.0803 ms 3.1418 ms 3.7969 ms]
NON_PRE_bench_bbf_commit_VECSZ=8192_OPNSZ=1024
time: [1.4319 s 1.4480 s 1.6174 s]
NON_PRE_bench_yinyan_commit_VECSZ=8192_OPNSZ=1024
time: [7.9894 s 15.607 s 18.245 s]
NON_PRE_bench_bbf_batch_open_VECSZ=8192_OPNSZ=1024
time: [3.2829 s 3.3400 s 3.5167 s]
NON_PRE_bench_yinyan_batch_open_VECSZ=8192_OPNSZ=1024
time: [2.4249 s 2.4395 s 2.6274 s]
NON_PRE_bench_bbf_verify_VECSZ=8192_OPNSZ=1024
time: [8.3548 ms 8.4648 ms 9.4972 ms]
NON_PRE_bench_yy_verify_VECSZ=8192_OPNSZ=1024
time: [5.0062 ms 5.2019 ms 6.3006 ms]
NON_PRE_bench_bbf_commit_VECSZ=16384_OPNSZ=2048
time: [2.9306 s 2.9483 s 3.1279 s]
NON_PRE_bench_yinyan_commit_VECSZ=16384_OPNSZ=2048
time: [16.122 s 33.912 s 40.085 s]
NON_PRE_bench_bbf_batch_open_VECSZ=16384_OPNSZ=2048
time: [6.5310 s 6.6796 s 7.0182 s]
NON_PRE_bench_yinyan_batch_open_VECSZ=16384_OPNSZ=2048
time: [4.6432 s 4.6465 s 4.6848 s]
NON_PRE_bench_bbf_verify_VECSZ=16384_OPNSZ=2048
time: [12.220 ms 12.382 ms 14.349 ms]
NON_PRE_bench_yy_verify_VECSZ=16384_OPNSZ=2048
time: [8.1466 ms 8.2486 ms 9.5713 ms]
NON_PRE_bench_bbf_commit_VECSZ=32768_OPNSZ=4096
time: [5.6557 s 5.6711 s 5.8354 s]
NON_PRE_bench_yinyan_commit_VECSZ=32768_OPNSZ=4096
time: [31.052 s 69.760 s 83.420 s]
NON_PRE_bench_bbf_batch_open_VECSZ=32768_OPNSZ=4096
time: [14.594 s 14.752 s 14.816 s]
NON_PRE_bench_yinyan_batch_open_VECSZ=32768_OPNSZ=4096
time: [9.2279 s 9.3159 s 10.402 s]
NON_PRE_bench_bbf_verify_VECSZ=32768_OPNSZ=4096
time: [25.852 ms 26.191 ms 30.239 ms]
NON_PRE_bench_yy_verify_VECSZ=32768_OPNSZ=4096
time: [22.544 ms 22.869 ms 26.552 ms]
NON_PRE_bench_bbf_commit_VECSZ=65536_OPNSZ=8192
time: [12.457 s 12.573 s 12.629 s]
NON_PRE_bench_yinyan_commit_VECSZ=65536_OPNSZ=8192
time: [69.865 s 177.93 s 215.47 s]
NON_PRE_bench_bbf_batch_open_VECSZ=65536_OPNSZ=8192
time: [36.651 s 36.714 s 36.849 s]
NON_PRE_bench_yinyan_batch_open_VECSZ=65536_OPNSZ=8192
time: [19.567 s 19.631 s 19.835 s]
NON_PRE_bench_bbf_verify_VECSZ=65536_OPNSZ=8192
time: [78.275 ms 79.200 ms 90.788 ms]
NON_PRE_bench_yy_verify_VECSZ=65536_OPNSZ=8192
time: [73.020 ms 74.411 ms 85.288 ms]
Gnuplot not found, disabling plotting
2. 库依赖
- num-bigint：为使用纯Rust语言实现的Big integer库。通过feature来配置，如
features=["rand"]与rand="0.7"结合可生成random big integers。
num-bigint = { version = "0.3", package = "num-bigint-dig", features = ["rand", "i128", "u64_digit", "serde", "prime"] }
类似的Big integer库有：
- num-traits：为Numeric traits for generic mathematics in Rust。
- num-integer：为Integer trait and functions for Rust。
- num-iter：为Generic Range iterators for Rust。
- rand：Rand provides utilities to generate random numbers, to convert them to useful types and distributions, and some randomness-related algorithms。
- failure：failure is designed to make it easier to manage errors in Rust，提供了2个核心元素：
– Fail：A new trait for custom error types。
– Error：A struct which any type that implementsFailcan be cast into。 - failure_derive：derives for the failure crate。实际未使用。
- blake2：Pure Rust implementation of the BLAKE2 hash function family。
- generic-array：数组库。implements generic array types for Rust。
- byteorder：provides convenience methods for encoding and decoding numbers in either big-endian or little-endian order。
- bitvec：bit位操作。bitvec enables Rust projects to have complete, bit-level, control of memory, with types that fit in with existing Rust idioms and patterns. The bit-precision pointers in this crate allow creation of more powerful bit-masks, set arithmetic, compact data storage, and I/O packet processing。
- rand_chacha：密码学安全的随机数生成器，使用ChaCha算法。A cryptographically secure random number generator that uses the ChaCha algorithm.
ChaCha is a stream cipher designed by Daniel J. Bernstein[^1], that we use as an RNG. It is an improved variant of the Salsa20 cipher family, which was selected as one of the “stream ciphers suitable for widespread adoption” by eSTREAM[^2]. - serde：Serde is a framework for serializing and deserializing Rust data structures efficiently and generically。
- classygroup （https://github.com/stichtingorganism/classygroup）：The Group of unknown order that is used to sample our Class groups of imaginary quadratic order。Class groups of binary quadratic forms omits the trusted setup that RSA needs.【实际可用ras_group或者classy group，默认为ras_group】
[features]
default = ["rsa_group"]
class_group = ["serde", "classygroup"]
rsa_group = []
- criterion：Statistics-driven Micro-benchmarking in Rust。Criterion.rs helps you write fast code by detecting and measuring performance improvements or regressions, even small ones, quickly and accurately. You can optimize with confidence, knowing how each change affects the performance of your code.
criterion的主要特征有：
– Statistics： Statistical analysis detects if, and by how much, performance has changed since the last benchmark run。
– Charts：支持使用gnuplot来生成detailed graphs of benchmark results。
– Stable-compatible：Benchmark your code without installing nightly Rust。
3. 代码结构
https://github.com/nicola/rust-yinyan 总体代码结构为：
-
benches：benchmark代码。
–accumulators_benchmarks.rs：为[BBF19]2018年论文《Batching Techniques for Accumulators with Applications to IOPs and Stateless Blockchains》算法benchmark code。
–vcs_benchmarks.rs：为2020年论文《Vector Commitment Techniques and Applications to Verifiable Decentralized Storage》中第一种方式构建的VCS算法benchmark code。 -
results：内涵plot.py脚本及一些结果数据。
-
src：算法核心源代码。
– bin：内含bench_exp.rs，custom_bench_client.rs和custom_bench_client_pre.rs执行文件，可分别就[BBF19]和本论文算法进行了性能对比。
– group：unknown order group实现。
– vc：对于[BBF19]的算法实现主要在binary.rs中，对于本论文的算法实现主要在yinyan.rs中。
接下来，将主要关注yinyan.rs中的代码实现。
4. protocol代码实现
https://github.com/nicola/rust-yinyan 中实际实现与上述略有差异，关注的relation为：
对应的
protocol non-interactive实现可表示为：
-
: run
，设置
。
其中Prover的输入为 ，Verifier的输入为 。 - Prover和Verifier均可计算prime ：
- Prove：
– 计算
– 计算
– 计算
– 整个proof内容为 - Verify：
– 计算
– Output 1 iff .
设置
，主要函数有：ni_poprod_prove和ni_poprod_verify。
pub type PoprodProof = (BigUint, BigUint, BigUint, BigUint);
/// NI-PoProd' Prove
pub fn ni_poprod_prove(
g: &BigUint,
h: &BigUint,
y1: &BigUint,
y2: &BigUint,
x1: &BigUint,
x2: &BigUint,
z: &BigUint,
n: &BigUint,
) -> PoprodProof {
// l <- H_prime(g, h, y1, y2)
let mut to_hash = g.to_bytes_be();
to_hash.extend(&h.to_bytes_be());
to_hash.extend(&y1.to_bytes_be());
to_hash.extend(&y2.to_bytes_be());
let l: BigUint = hash_prime::<_, Blake2b>(&to_hash).into();
// (q1, q2, q3) = (x1/l, x2/l, z/l)
// (r1, r2) = (x1 mod l, x2 mod l)
let (q1, r1) = x1.div_rem(&l);
let (q2, r2) = x2.div_rem(&l);
let (q3, _) = z.div_rem(&l);
// (Q1, Q2, r1, r2, r3) = (h^q1, h^q2 * g^q3)
let q_big1 = h.modpow(&q1, n);
let q_big2 = h.modpow(&q2, n) * &g.modpow(&q3, n) % n;
(q_big1, q_big2, r1, r2) as PoprodProof
}
/// NI-PoProd3 Verify
pub fn ni_poprod_verify(
g: &BigUint,
h: &BigUint,
y1: &BigUint,
y2: &BigUint,
pi: &PoprodProof,
n: &BigUint,
) -> bool {
let (q_big1, q_big2, r1, r2) = pi;
let mut to_hash = g.to_bytes_be();
to_hash.extend(&h.to_bytes_be());
to_hash.extend(&y1.to_bytes_be());
to_hash.extend(&y2.to_bytes_be());
let l: BigUint = hash_prime::<_, Blake2b>(&to_hash).into();
// r1, r2 < l
let range_check = r1 < &l && r2 < &l;
// r3 = r1 * r2 mod l
let r3 = (r1 * r2) % &l;
// Q1^l h^r1 = y1
let q1_check = {
let lhs = (q_big1.modpow(&l, n) * h.modpow(&r1, n)) % n;
lhs == *y1
};
// Q2^l * h^r2 * g^r3 = y2
let q2_check = {
let tmp = q_big2.modpow(&l, n) * h.modpow(&r2, n) % n;
let lhs = tmp * g.modpow(&r3, n) % n;
lhs == *y2
};
range_check && q1_check && q2_check
}
实例example为：
let n = rng.gen_biguint(1024);
let g = rng.gen_biguint(1024) % &n;
let h = rng.gen_biguint(1024) % &n;
let x1 = rng.gen_biguint(128);
let x2 = rng.gen_biguint(128);
let z = &x1 * &x2;
// h^x1
let y1 = h.modpow(&x1, &n);
// h^x2 * g^z
let y2 = h.modpow(&x2, &n) * g.modpow(&z, &n) % &n;
let pi = ni_poprod_prove(&g, &h, &y1, &y2, &x1, &x2, &z, &n);
assert!(ni_poprod_verify(&g, &h, &y1, &y2, &pi, &n));
5. 第一种SVC实现——基于RSA accumulators
5.1 基础结构：
#[derive(Clone, Debug)]
pub struct Config {
pub lambda: usize,
pub k: usize, //vector内每个元素以k-bit 二进制表示。
pub n: usize, //对rsa_group来说，该值不能低于64.详见ras.rs。
pub size: usize, //待commit的vector size
pub precomp_l: usize,
pub ph: Rc<PrimeHash>
}
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Debug, Clone)]
pub struct YinYanVectorCommitment<'a, A: 'a + UniversalAccumulator + BatchedAccumulator + FromParts>
{
lambda: usize, // security param
k: usize, // word size
size: usize, // max words in the vector
uacc: A,
pub accs: Vec<(A, A)>, // lenght of accs must be k
_a: PhantomData<&'a A>,
prod_proofs: Vec<proofs::PoprodProof>,
modulus: BigUint,
precomp_l: usize, // each precomputed proof refers to a chunk of size precomp_l
precomp_N: usize, // There are precomp_N chunks (and precomputed proofs)
pi_precomp: Vec<Proof>, // precomputed proofs
hash: Rc<PrimeHash>
}
5.2 VC.Setup：
fn setup<G, R>(rng: &mut R, config: &Self::Config) -> Self
where
G: PrimeGroup,
R: CryptoRng + Rng,
{
let (modulus, g) = G::generate_primes(rng, config.n).unwrap();
let two = BigUint::from_u64(2 as u64).unwrap();
let precomp_N = if config.precomp_l !=0 {config.size/config.precomp_l} else {0};
YinYanVectorCommitment {
lambda: config.lambda,
k: config.k,
size: config.size,
modulus: modulus.clone(),
prod_proofs: vec![],
uacc: A::from_parts(modulus.clone(), rng.gen_biguint(config.n)),
accs: (0..config.k)
.map(|_| {
let r0 = rng.gen_biguint(config.n);
let r1 = rng.gen_biguint(config.n);
//let g = tmp.modpow(&two, &modulus);
(
A::from_parts(modulus.clone(), g.modpow(&r0, &modulus)),
//A::from_parts(modulus.clone(), g.modpow(&r1, &modulus)),
//代码中此处需修改，因为PoProd protocol实际实现时g_1=g_2
A::from_parts(modulus.clone(), g.modpow(&r0, &modulus)),
)
})
.collect(),
_a: PhantomData,
precomp_l: config.precomp_l,
precomp_N: precomp_N,
pi_precomp: Vec::with_capacity( precomp_N ), // size is precomp_N
hash: Rc::clone(&config.ph),
}
}
5.3 VC.Specialize：
pub fn specialize(&mut self, size: usize) -> &BigUint {
// TODO: if already specialized skip first part
for i in 0..size {
// TODO: eventually do batchadd (check how we do it in commit)
self.uacc.add(&self.hash.get(i));
}
self.uacc.state()
}
fn state(&'a self) -> Self::State {
self.accs
.iter()
.map(|acc| (acc.0.state(), acc.1.state()))
.collect()
}
- VC.Com：
fn commit(&mut self, words: &[Self::Domain]) -> Self::Commitment {
// i = 0..m (m number of words)
for (i, v) in words.iter().enumerate() {
debug_assert!(v.len() == self.k);
debug_assert!(i < self.hash.max_sz);
// p_i
let prime = self.hash.get(i);
// j = 0..k (k number of bits in each word)
// TODO: can be done with batch add!
for (bit, acc) in v.iter().zip(self.accs.iter_mut()) {
if *bit {
// B_j
acc.1.add(&prime);
} else {
// A_j
acc.0.add(&prime);
}
}
}
let g = self.uacc.g();
let (U_n, u) = (self.uacc.state(), self.uacc.set());
self.prod_proofs = self
.accs
.iter()
.map(|acc| {
let g_j = acc.0.g();
debug_assert!(g_j == acc.1.g());
let (A_j, a_j) = (acc.0.state(), acc.0.set());
let (B_j, b_j) = (acc.1.state(), acc.1.set());
let pi =
proofs::ni_poprod_prove(
g,
g_j,
A_j,
&((B_j * U_n) % &self.modulus),
a_j,
b_j,
u,
&self.modulus,
);
pi
})
.collect();
Self::Commitment {
states: self
.state()
.iter()
.map(|acc| (acc.0.clone(), acc.1.clone()))
.collect(),
prods: self.prod_proofs.clone(),
}
}
5.4 VC.Open：
fn open(&self, wd: &Self::Domain, i: usize) -> Self::Proof {
let p_i = self.hash.get(i);
let proof: Proof = wd
.iter().enumerate()
.zip(self.accs.iter())
.map( |((idx, bit), acc)| {
let pf_bit = if *bit {
acc.1.mem_wit_create(&p_i)
} else {
acc.0.mem_wit_create(&p_i)
};
self.mk_triv_pf(*bit, idx, pf_bit)
} )
.collect();
proof
}
5.5 VC.Ver：
fn verify(&self, wd: &Self::Domain, i: usize, pi: &Self::Proof) -> bool {
let p_i = self.hash.get(i);
// Make sure proof is of the right size
if self.prod_proofs.len() != self.k || pi.len() != self.k {
return false;
}
// Verify accumulator proof
let accs_check = wd
.iter()
.zip(self.accs.iter())
.zip(pi.iter())
.all(|((bit, acc), w)| {
self.bit_verify(acc, bit, w, &p_i)
});
// Verify product proof
let uacc_check = {
let g = self.uacc.g();
let U_n = self.uacc.state();
self.accs
.iter()
.zip(self.prod_proofs.iter())
.all(|(acc, prod)| {
let g_j = acc.0.g();
let (A_j, B_j) = (acc.0.state(), acc.1.state());
proofs::ni_poprod_verify(
g, // g
g_j, // h
A_j, // y1
&((B_j * U_n) % &self.modulus), // y2
prod, // pi
&self.modulus,
)
})
};
accs_check && uacc_check
}
5.6 VC.Agg：
fn aggregate_proofs(
&self,
pf1:&Vec<(BigUint, BigUint)>, vals1:&[Domain], I1:&[usize],
pf2:&Vec<(BigUint, BigUint)>, vals2:&[Domain], I2:&[usize]
) -> Vec<(BigUint, BigUint)> {
let mut pfs:Vec<(BigUint, BigUint)> = vec![];
for i in 0..self.k {
let vals1_i = vals1.iter().map(|v| v[i]).collect();
let vals2_i = vals2.iter().map(|v| v[i]).collect();
let pf_i = self.aggregate_proofs_bit(
&pf1[i], &vals1_i, I1, &pf2[i], &vals2_i, I2);
pfs.push(pf_i);
}
pfs
}
// This version is not for generic words but only for case k = 1 for now
fn aggregate_proofs_bit(
&self,
pf1:&(BigUint, BigUint), vals1:&Vec<bool>, I1:&[usize],
pf2:&(BigUint, BigUint), vals2:&Vec<bool>, I2:&[usize]
) -> (BigUint, BigUint) {
self.aggregate_proofs_bit_primes(
pf1, vals1, &to_primes(&self.hash, &I1.to_vec()), pf2, vals2, &to_primes(&self.hash, &I2.to_vec()))
}
// This version is not for generic words but only for case k = 1 for now
fn aggregate_proofs_bit_primes(
&self,
pf1:&(BigUint, BigUint), vals1:&Vec<bool>, pI1:&[BigUint],
pf2:&(BigUint, BigUint), vals2:&Vec<bool>, pI2:&[BigUint]
) -> (BigUint, BigUint) {
// NB: We assume pI1 and pI2 are disjoint
let pp1 = partitioned_prime_prod_primes(vals1, pI1);
let pp2 = partitioned_prime_prod_primes(vals2, pI2);
self.aggregate_proofs_bit_part_prod(pf1, pp1, pf2, pp2)
}
// This version is not for generic words but only for case k = 1 for now
fn aggregate_proofs_bit_part_prod(
&self,
pf1:&(BigUint, BigUint), part_prod1:(BigUint, BigUint),
pf2:&(BigUint, BigUint), part_prod2:(BigUint, BigUint)
) -> (BigUint, BigUint) {
// NB: We assume pI1 and pI2 are disjoint
let (a1, b1) = part_prod1;
let (a2, b2) = part_prod2;
let pf_zero = shamir_trick(&pf1.0, &pf2.0, &a1, &a2, &self.modulus).unwrap();
let pf_one = shamir_trick(&pf1.1, &pf2.1, &b1, &b2, &self.modulus).unwrap();
(pf_zero.clone(), pf_one.clone())
}
5.7 VC.Disagg：
参考资料：
[1] https://github.com/rust-num/num-bigint