1. The interior point method is to search inside the feasible region, and finally converge to the optimal solution boundary
2. Commonly used interior point methods include affine scale method, logarithmic barrier method and primitive dual method
In addition to the simplex method and dual theory to solve the linear programming (LP) problem, there is also a search solution - the interior point method (interior point method) , which moves inside the feasible region. Today we will learn three interior point methods, including: affine-scaling , log-barrier and primal-dual.
Again, for ease of introduction, we introduce a new example - Franny's firewood:
Every year Franny sells 3 matches from her grove. One potential customer is willing to pay $90 for a half stick, another $150 for a stick. Our question is how much firewood should Franny sell to each customer to maximize her profit? Assume that each customer can buy as much as possible.
Modeling the above scenario, it is easy to get:
Draw its feasible region area on the two-dimensional coordinate axis.
0
The logic of the interior point method
The interior point method is an algorithm that performs search iterations inside the feasible region , and it has a significant advantage: no constraints work, and all directions are feasible . But we know that the maximum improvement direction of the maximization model is the gradient direction, and the maximum improvement direction of the minimization model is the negative gradient direction. In this way, the interior point method will inevitably stop at the boundary point. Assuming the Franny wood problem, improve from the interior point X0(1,0.5) to the gradient direction, according to the principle of maximum step size, the next point will stay on the boundary of the feasible region X1:
Therefore, the key to the interior point method is whether the search point is always kept in the "middle" of the feasible region to guide the optimal solution to be found, which must follow: the interior point method starts from an internal feasible point, and between a series of internal points Search until the optimal solution converges to the boundary of the feasible region.
To check whether each iteration point is an interior point, we first need to change the LP problem into a standard linear programming, taking the Franny firewood problem as an example:
In this way, it is easy to judge whether a feasible solution is an interior point: given a feasible solution of the linear programming standard form, if each component of the feasible solution is strictly positive, then the feasible solution is an interior point
Here is an example to illustrate:
It can be seen that for the standard LP problem, the interior point must satisfy: the constraint condition Ax=b holds; the component is strictly greater than 0. The direction of its next search point should satisfy:
According to the projection theory , we can get the moving direction of the interior point:
Generally speaking, d is the positive and negative gradient direction, because this direction is the fastest improving direction, and Δx guarantees the feasible direction. The above theory states that the direction of movement is as close as possible to the direction of improvement and remains on the interior point. So is Δx the direction of improvement? The answer is yes. We can prove:
for example:
To sum up, the core of the interior point method is to avoid the boundary of the feasible domain before finding the optimal solution, and to ensure that each round of iteration is a feasible improvement direction.
1
Affine Scaling Algorithm
According to the principle of the previous section, we propose an effective tool to avoid the boundary - scaling, first define the diagonal matrix form of the current solution:
Because the components of interior points are strictly greater than 0, we construct affine scaling and inverse affine scaling:
After transformation, the form of the original standard linear programming will be changed :
Take the Franny firewood problem as an example, c=(90,150,0), A=(0.5,1,1), b=(3), then:
The form after the affine scale transformation, to get the next search point, according to the projection theory, we get:
The feasible improvement direction Δx of the original objective function can be obtained by using the inverse affine scale transformation:
Then there is the last question, what is the step size of the move:
Obviously, we must strictly ensure that each component of the solution obtained at each step is strictly greater than 0 until it converges to the optimal solution. For the maximization problem, if Δx is non-negative, then the corresponding standard linear programming model is unbounded. otherwise:
Why is this, because the y after the affine scale transformation is all 1:
It shows that the search range is carried out within a unit circle (dimension n), and it is only necessary to realize the affine scale transformation by finding the limit of this sphere.
In summary, we get the steps of the affine scaling algorithm :
Taking the Franny firewood problem as an example, let's take a look at the details of the affine scaling algorithm:
It can be seen that when the iteration reaches the ninth time, the error is very close to the optimal solution. We can easily implement this process using the python program:
The following 3D diagram shows the search process (only the first 6 iterations are shown):
The two-dimensional diagram more precisely shows that the search process is carried out at the interior point of the feasible region:
Note: the code is placed in the appendix
2
logarithmic handicap
Next we introduce the second interior point method - the logarithmic barrier method. First give the model directly:
The idea here is to make the search point principle boundary as much as possible. Why is this because:
Take the Franny firewood problem as an example:
It can be seen that the closer to the boundary, the more significant the impact of the obstacle item on the real value, thus avoiding the search algorithm from touching the boundary of the feasible region during iteration.
The planning problem with logarithmic barriers is nonlinear. We use the idea of Newton's method to iterate the second-order Taylor expansion on the multivariate function. The derivation process is no longer expanded. Here we directly give the direction of movement:
The step size of the obstacle method from Newton iteration is:
In the above formula, the multiple positioning is 0.9 because due to the non-linearity of the target, the search process may stop improving before approaching the maximum step size, or it may start to reduce the value of the objective function after a larger step size near the current solution.
Next, the last problem remains—the determination of the obstacle factor. Obviously, when the factor is large, it has a significant effect on hindering the interior points close to the boundary, and when it is small, it encourages the search process to approach the boundary. Therefore, the barrier algorithm usually starts with a large factor and reduces it to 0 as the search progresses slowly
Based on the above, we get the steps of the obstacle algorithm:
Still taking the Franny firewood problem as an example, implement this process with code. The following figure shows the first five two-dimensional search processes:
The code developed by the editor went through 12 iterations to reach the optimum. The following are some screenshots. See the appendix for the code.
3
Primal Duality
In this section we discuss the last algorithm of the interior point method - the primal dual interior point method . In the dual theory, we get the form of the standard original problem and the dual problem and the relaxation conditions:
Portal: A dual theory of linear programming
We add the slack variable to the dual problem, transform it into a form without inequality constraints, and get:
At the optimal solution , we have:
If the solution is not optimal, the above equation is not satisfied, and the primal-dual interior-point search iterates using their duality gap :
The iterative principle is: the original dual interior point method always keeps the solutions of the original problem and the dual problem strictly feasible in each iteration, and systematically reduces the degree of violation of complementary slackness during the search process
Specifically, the primal dual interior point method sets the target μ>0 for each violation degree in the search, finds a feasible moving direction that can reduce the difference between the current complementary relaxation result and the target, and gradually reduces μ to 0 to achieve complementary relaxation ,this needs:
Then the last problem remains: the determination of the step size λ. The algorithm keeps the search process within the feasible region by adjusting a positive barrier factor (boundary prevention) - constant δ
In summary, we get the steps of the original dual interior point method:
We will not implement this algorithm here, and will combine the dual theory with the implementation code in the future! Let's end today. Happy National Day! !
【appendix】:
#放射尺度变换
import sys
import numpy as np
c = np.array([90,150,0])
A = np.array([0.5,1,1])
# D = []
t = 0
x_t = np.array([1,0.5,2])
while True:
# D.append(list(x_t))
print('t={}时的目标取值:{}'.format(t,np.dot(c,x_t)))
for i in list(x_t):
if round(i,6) == 0:
print('迭代结束,最优解为{}'.format(np.around(x_t,4)))
sys.exit()
X_t = np.diag(x_t)
c_t = np.dot(c,X_t)
A_t = np.dot(A,X_t)
P_t = np.identity(3) - np.dot(A_t.reshape(3,1),A_t.reshape(1,3))/np.dot(A_t,A_t.T)
delat_x = np.dot(X_t,np.dot(P_t,c_t))
print('移动方向:{}'.format(np.around(delat_x,2)))
a = np.dot(delat_x,np.linalg.inv(X_t))
lam = 1/np.sqrt(np.dot(a,a))
print('步长大小：{:.5f}'.format(lam))
x_t = x_t + lam*delat_x
print('下一搜索点:{}'.format(np.around(x_t,4)))
t += 1
# D.append(list(delat_x))
print('*****************************************')#对数障碍法
import numpy as np
import pandas as pd
c = np.array([90,150,0])
A = np.array([0.5,1,1])
D = []
t = 0
x_t = np.array([1,0.5,2])
y = 0
u = 16
while True:
D.append(list(x_t))
y_old = np.dot(c, x_t)+u*np.sum(np.log(x_t))
print('t={}时的实际目标值:{}'.format(t,np.dot(c,x_t)))
print('t={}时的障碍目标值:{}'.format(t, y_old))
X_t = np.diag(x_t)
c_t = np.dot(c,X_t)
A_t = np.dot(A,X_t)
P_t = np.identity(3) - np.dot(A_t.reshape(3,1),A_t.reshape(1,3))/np.dot(A_t,A_t.T)
delat_x = np.dot(np.dot(X_t,P_t),c_t+u)
print('移动方向:{}'.format(np.around(delat_x,3)))
lam_list = []
for i in range(3):
if delat_x[i] < 0:
lam_list.append(-0.9*x_t[i] / delat_x[i])
lam = min(1/u,min(lam_list))
print('障碍因子为:{},步长大小：{:.5f}'.format(u,lam))
x_t = x_t + lam*delat_x
print('下一搜索点:{}'.format(np.around(x_t,4)))
y_new = np.dot(c, x_t)+u*np.sum(np.log(x_t))
t += 1
if y_new - y_old < 1:
if u < 0.5:
print('迭代结束,最优解为{}'.format(np.around(x_t, 2)))
break
else:
u = u/2
else:
continue
# D.append(list(delat_x))
print('*****************************************')import matplotlib.pyplot as plt
plt.rcParams['font.sans-serif']=['SimHei']
plt.rcParams['axes.unicode_minus']=False
fig = plt.figure(figsize=(12,6))
ax = plt.axes(projection='3d')
ax.scatter(A['x1'],A['x2'],A['x3'],color='red',s=25)
ax.quiver(A['x1'],A['x2'],A['x3'],A['u'],A['v'],A['w'],arrow_length_ratio=0.2,color='black')
ax.set_xlabel('x1')
ax.set_ylabel('x2')
ax.set_zlabel('x3')
ax.set_xlim(0,6)
ax.set_ylim(0,3)
ax.set_zlim(0,2)
ax.set_title('弗兰妮木柴问题搜索过程')
plt.show()plt.figure(figsize=(6,4))
plt.fill_between(np.linspace(0,6,500),np.tile(np.array(0),500),3-0.5*np.linspace(0,6,500),facecolor="#DCDCDC")
plt.scatter(A['x1'],A['x2'],color='black',linewidths=0.5)
# plt.text(-6,-1,s='X0(-5,0)',verticalalignment='bottom')
for _,row in A.iterrows():
plt.arrow(row.x1, row.x2, row.u, row.v, length_includes_head=True, head_width=0.1, width=0.02, alpha=0.1, fc='#CD2626', ec='blue')
plt.xlim(0,6)
plt.ylim(0,3)
plt.xlabel("x1")
plt.ylabel("x2")
plt.title("弗兰妮木柴问题二维搜索过程")
plt.tight_layout()
plt.show()