Django Restfarmwork query interface optimization

1. Avoid database query in a for loop

The relevant code examples:

# 优化前写法:
students = get_training_team_student_info(data.get('resource_id'))

for student in students:
    student_name = student.get('user_name')
    # 获取参训人员分数
    student_score_obj = get_training_team_score_first(
            training_team=data.get('resource_id'), 
            user=student.get('user_id')
            )
    if not student_score_obj:
        student_score = 0
    else:
        student_score = student_score_obj

# 优化后写法:
students = get_training_team_student_info(data.get('resource_id'))
# 查出所有学员成绩数据
student_score_lists = get_training_team_score(training_team=data['resource_id'])
student_score_dict = {}
for student_score in student_score_lists:
    student_score_dict[student_score.user_id] = student_score.final_score

for student in students:
    student_name = student.get('user_name')
    # 获取参训人员分数
    student_score_obj = student_score_dict.get(student.get('user_id'))

    if not student_score_obj:
        student_score = 0
    else:
        student_score = student_score_obj

Description: first find the actual situation consists of all data mapping relationship, a value for loop operation

2. For values_list, obtaining a query result field method, using the list to generate a list of field derivation of formula

The relevant code examples:

# 优化前相关代码:
# 获取角色下已下发的任务hash列表
send_missions = model_api.get_training_team_mission(
    training_team=training_team_hash,
    training_role=training_role.resource_id
).values_list('mission_resource_id', flat=True)

"""优化前SQL执行
此sql对应上面代码的 send_missions = model_api.get_training_team_mission(training_team=training_team_hash,training_role=training_role.resource_id)
SELECT `ce_training_team_trainingteammission`.`id`, `ce_training_team_trainingteammission`.`resource_id`, `ce_training_team_trainingteammission`.`training_team`, `ce_training_team_trainingteammission`.`training_role`, `ce_training_team_trainingteammission`.`mission_resource_id`, `ce_training_team_trainingteammission`.`user_id`, `ce_training_team_trainingteammission`.`mission_status`, `ce_training_team_trainingteammission`.`user_answer`, `ce_training_team_trainingteammission`.`release_time`, `ce_training_team_trainingteammission`.`reporting_time`, `ce_training_team_trainingteammission`.`start_mission_time`, `ce_training_team_trainingteammission`.`status` FROM `ce_training_team_trainingteammission` WHERE (NOT (`ce_training_team_trainingteammission`.`status` = 0) AND `ce_training_team_trainingteammission`.`training_role` = 'ce66d38a3b0784ddd9074cecbc9bc288' AND `ce_training_team_trainingteammission`.`training_team` = 'ceb99b9d1677016bd09b38295422ee89'); args=(0, 'ce66d38a3b0784ddd9074cecbc9bc288', 'ceb99b9d1677016bd09b38295422ee89')

此sql对应上面代码的.values_list('mission_resource_id', flat=True)
SELECT `ce_training_team_trainingteammission`.`mission_resource_id` FROM `ce_training_team_trainingteammission` WHERE (NOT (`ce_training_team_trainingteammission`.`status` = 0) AND `ce_training_team_trainingteammission`.`training_role` = 'ce66d38a3b0784ddd9074cecbc9bc288' AND `ce_training_team_trainingteammission`.`training_team` = 'ceb99b9d1677016bd09b38295422ee89')  LIMIT 21; args=(0, 'ce66d38a3b0784ddd9074cecbc9bc288', 'ceb99b9d1677016bd09b38295422ee89')

执行get_training_team_mission()函数时会进行一次数据库查询,执行.values_list时会再一次进行数据库查询
"""
# 优化后相关代码:
send_missions = model_api.get_training_team_mission(
    training_team=training_team_hash,
    training_role=training_role.resource_id
)
mission_resource_id_list = [mission.mission_resource_id for mission in send_missions]
"""优化前SQL执行
SELECT `ce_training_team_trainingteammission`.`id`, `ce_training_team_trainingteammission`.`resource_id`, `ce_training_team_trainingteammission`.`training_team`, `ce_training_team_trainingteammission`.`training_role`, `ce_training_team_trainingteammission`.`mission_resource_id`, `ce_training_team_trainingteammission`.`user_id`, `ce_training_team_trainingteammission`.`mission_status`, `ce_training_team_trainingteammission`.`user_answer`, `ce_training_team_trainingteammission`.`release_time`, `ce_training_team_trainingteammission`.`reporting_time`, `ce_training_team_trainingteammission`.`start_mission_time`, `ce_training_team_trainingteammission`.`status` FROM `ce_training_team_trainingteammission` WHERE (NOT (`ce_training_team_trainingteammission`.`status` = 0) AND `ce_training_team_trainingteammission`.`training_role` = 'ce66d38a3b0784ddd9074cecbc9bc288' AND `ce_training_team_trainingteammission`.`training_team` = 'ceb99b9d1677016bd09b38295422ee89'); args=(0, 'ce66d38a3b0784ddd9074cecbc9bc288', 'ceb99b9d1677016bd09b38295422ee89')
 只会在执行get_training_team_mission()函数时时进行一次数据库查询
"""

3. In the page data acquisition interface list, only returns a list of pages that need data to reduce unnecessary data query

The relevant code examples:

# 优化前返回字段:
fields = ("resource_id", "name", "description", "category", "pre_skill", "train_skill", "start_time", "end_time",
            "template", "scene", "difficult", "scene_config_data", "training_role_data", "duration_time",
            "scene_config", "scene_start_time", "scene_end_time", "training_status", "scene_data", "roles_data")

# 优化后针对列表页接口返回指定字段
def get_serializer(self, *args, **kwargs):
    """
    获取序列化器

    :param args: 位置参数
    :param kwargs: 不定长字典参数
    :return: 序列化器
    """
    if self.action == 'list':
        kwargs['fields'] = (
            "resource_id", "name", "description", "category", "pre_skill", "train_skill", "start_time", "end_time",
            "template", "difficult", "scene_config", "training_role_data", "duration_time",
            "scene_start_time", "scene_end_time", "training_status"
        )
    return super().get_serializer(*args, **kwargs)

4. For many-related queries, using prefetch_related preloading

The relevant code examples:

# 优化前相关代码:
role_list = get_training_role(
        training_team=data['resource_id']
    ).exclude(users__id=user_id)

for role in role_list:
    role_name = role.name
    users = role.users.all()
    user_names = [user.username for user in users]
    role_info_list.append({
        'role_name': role_name,
        'users': user_names
    })
"""优化前执行sql
此sql对应上面代码的 role_list = get_training_role(training_team=data['resource_id'])
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE `ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54'; args=('a0b8cda8d219f5a4af005e73c542ef54',)

此sql对应上面代码的 .exclude(users__id=user_id)
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE (`ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54' AND NOT (`ce_training_team_trainingrole`.`id` IN (SELECT U1.`trainingrole_id` FROM `ce_training_team_trainingrole_users` U1 WHERE U1.`user_id` = 7))); args=('a0b8cda8d219f5a4af005e73c542ef54', 7)

此sql对应上面代码的 users = role.users.all()
SELECT `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` = 235; args=(235,)
SELECT `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` = 236; args=(236,)
SELECT `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` = 237; args
"""

# 优化后相关代码
role_list = model_api.get_training_role(
        training_team=data['resource_id']
    ).exclude(users__id=user_id).prefetch_related('users')

for role in role_list:
    role_name = role.name
    users = role.users.all()
    user_names = [user.username for user in users]
    role_info_list.append({
        'role_name': role_name,
        'users': user_names
    })

"""优化后执行sql
此sql对应上面代码的 role_list = get_training_role(training_team=data['resource_id'])
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE `ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54'; args=('a0b8cda8d219f5a4af005e73c542ef54',)

此sql对应上面代码的 .exclude(users__id=user_id)
SELECT `ce_training_team_trainingrole`.`id`, `ce_training_team_trainingrole`.`resource_id`, `ce_training_team_trainingrole`.`training_team`, `ce_training_team_trainingrole`.`name`, `ce_training_team_trainingrole`.`description`, `ce_training_team_trainingrole`.`role_group`, `ce_training_team_trainingrole`.`missions_scheme`, `ce_training_team_trainingrole`.`nodes` FROM `ce_training_team_trainingrole` WHERE (`ce_training_team_trainingrole`.`training_team` = 'a0b8cda8d219f5a4af005e73c542ef54' AND NOT (`ce_training_team_trainingrole`.`id` IN (SELECT U1.`trainingrole_id` FROM `ce_training_team_trainingrole_users` U1 WHERE U1.`user_id` = 7))); args=('a0b8cda8d219f5a4af005e73c542ef54', 7)

此sql对应上面代码的 .prefetch_related('users')
SELECT (`ce_training_team_trainingrole_users`.`trainingrole_id`) AS `_prefetch_related_val_trainingrole_id`, `sv_auth_user`.`id`, `sv_auth_user`.`password`, `sv_auth_user`.`last_login`, `sv_auth_user`.`is_superuser`, `sv_auth_user`.`username`, `sv_auth_user`.`first_name`, `sv_auth_user`.`last_name`, `sv_auth_user`.`email`, `sv_auth_user`.`is_staff`, `sv_auth_user`.`is_active`, `sv_auth_user`.`date_joined`, `sv_auth_user`.`resource_id`, `sv_auth_user`.`logo`, `sv_auth_user`.`nickname`, `sv_auth_user`.`name`, `sv_auth_user`.`organization_id`, `sv_auth_user`.`status` FROM `sv_auth_user` INNER JOIN `ce_training_team_trainingrole_users` ON (`sv_auth_user`.`id` = `ce_training_team_trainingrole_users`.`user_id`) WHERE `ce_training_team_trainingrole_users`.`trainingrole_id` IN (235, 236, 237); args=(235, 236, 237)

相对来说此例子只是少了一个sql,但是如果这个角色下的人员有很多的话,每一个人都会进行一次sql查询,
而优化后不管有多少人,只是进行了一次sql查询
"""

Development: Related select_related () and prefetch_related () operation can refer to: https://www.cnblogs.com/tuifeideyouran/p/4232028.html

The addition of as little as possible other operations after the cache function, e.g. .first () ,. exclude () ,. values_list () ,. values ​​() and other related operations

The relevant code examples:

# 优化后相关代码
@func_cache(ce_training_team_teacher_cache)
def get_training_role(**kwargs):
    """
    获取团队训练实战角色
    :param kwargs:参数
    :return: 团队训练实战角色obj
    """
    return TrainingRole.objects.filter(**kwargs)


@func_cache(ce_training_team_teacher_cache)
def get_training_role_first(**kwargs):
    """
    获取团队训练实战角色
    :param kwargs:参数
    :return: 团队训练实战角色obj
    """
    return TrainingRole.objects.filter(**kwargs).first()


@func_cache(ce_training_team_teacher_cache)
def get_training_role_prefetch(**kwargs):
    """
    获取团队训练实战角色预加载角色信息
    :param kwargs: 参数
    :return: 团队训练实战角色obj
    """
    return TrainingRole.objects.filter(**kwargs).prefetch_related('users')

Python by negative logic as before unnecessary database query conditions, e.g. values_list before () fetching a single data field, may be used instead of a for loop.
In the case of the use of more, a single cache may be provided for the function conditions, the above example get_training_role_prefetch () function.
each add one additional operation time will increase sql query for this feature, particularly to be adjusted according to actual code