基于典型国产图数据库的亿级数据高效导入与索引关联技术

    Efficient import and index association technology of 100 million level data based on typical domestic graph database

    • 摘要: 图数据库作为知识图谱的存储应用系统,擅长处理数据间的复杂关系和连接性问题,高效地查询和分析实体之间的关联性。NebulaGraph作为开源国产图数据库的典型代表,具有分布式架构和线性扩容特点,擅长处理千亿个顶点和万亿条边的超大规模数据集。但是,NebulaGraph在处理大规模数据导入时,却面临着操作流程独立、割裂、难以串行等挑战,由于人工操作各流程衔接不及时,导致数据导入与索引关联过程耗时较长,严重制约了其在大数据场景下的应用效率和用户体验。本文提出了一种适用于国产图数据库NebulaGraph的数据标准,并基于该标准实现了图空间构建、本体设计、数据导入、索引创建、数据关联等操作流程一体化,极大简化数据导入流程,显著提升数据导入效率,有力支撑大规模图谱自动化创建。此外,针对导入的海量数据,通过自动识别其实体、属性和关系,动态生成合理的图谱结构,不仅节省了人工构建图空间的时间,而且确保了图谱结构与实际数据的高度契合,增强了图谱分析的准确性和有效性。实验证明,针对亿级规模数据导入NebulaGraph的场景,通过人机交互构建图谱的方法无法完成数据导入,通过访问图数据库逐条导入数据的方法耗时过长,而本文提出的方法与上述方法相比,亿级规模数据导入速度从每秒596条提升到每秒14 331条,总体耗时降低95.8%,显著减少了数据导入的时间成本,提升了国产图数据库在大数据场景下的应用价值。

       

      Abstract: As a storage and application system for knowledge graphs, graph databases excel in handling complex relationships and connectivity among data, enabling efficient search and analysis of entity associations. NebulaGraph, recognized as a typical representative of open-source domestic graph database, possesses a distributed architecture and linear scalability, making it adept at processing ultra-large datasets consisting of hundreds of billions of vertices and trillions of edges. However, when handling large-scale data imports, challenges arise due to independent, fragmented, and difficult-to-serialize operational processes. As a result of inefficient manual operation and poor coordination between various processes, the data import and index association processes become time-consuming, severely constraining application efficiency and user experience in big data scenarios. A kind of data standard applicable to the domestic graph database NebulaGraph is proposed. Based on this standard, the integration of graph space construction, ontology design, data import, index creation, and data association is achieved, significantly simplifying the data import process and substantially enhancing the efficiency of data import. This approach supports the automated creation of large-scale graphs effectively. Moreover, by automatically identifying entities, attributes, and relationships within the imported massive data, a reasonable graph structure can be dynamically generated. This method not only saves the time required for manual graph space construction but also ensures a high degree of alignment between the graph structure and the actual data, thereby enhancing the accuracy and effectiveness of graph analysis. Experiments demonstrate that, for the scenario of importing gigabyte-scale data into NebulaGraph, methods involving human-computer interaction fail to complete data import, while methods that import data sequentially via access to the graph database are excessively time-consuming. In contrast, the proposed method increases the import speed from 596 records per second to 14 331 records per second for 100 million-scale data, reducing the overall processing time by 95.8%. This significantly reduces the time cost of data import and enhances the application value of domestic graph databases in big data scenarios.

       

    /

    返回文章
    返回