网站开发公司属于什么行业,大丰做网站,变色龙app制作平台,广告设计公司深圳营销策划公司对于RAG来说#xff0c;从文档中提取信息是一种不可避免的场景#xff0c;确保从源文件中提取出有效的内容对于提高最终输出的质量至关重要。 文件解析过程在RAG中的位置如图1所示#xff1a; 在实际工作中#xff0c;非结构化数据比结构化数据丰富得多。如果这些海量数据无… 对于RAG来说从文档中提取信息是一种不可避免的场景确保从源文件中提取出有效的内容对于提高最终输出的质量至关重要。 文件解析过程在RAG中的位置如图1所示 在实际工作中非结构化数据比结构化数据丰富得多。如果这些海量数据无法解析它们的巨大价值将无法实现。在非结构化数据中PDF文档占大多数。有效地处理PDF文档还可以极大地帮助管理其他类型的非结构化文档。 本文主要介绍解析PDF文件的方法为有效解析PDF文档和提取尽可能多的有用信息提供了算法和参考。
一、解析PDF的挑战 PDF文档是非结构化文档的代表然而从PDF文档中提取信息是一个具有挑战性的过程。 将PDF描述为输出指令的集合更准确而不是数据格式。PDF文件由一系列指令组成这些指令指示PDF阅读器或打印机在屏幕或纸张上显示符号的位置和方式这与HTML和docx等文件格式形成对比后者使用p、w:p、table和w:tbl等标记来组织不同的逻辑结构如图2所示 解析PDF文档的挑战在于准确提取整个页面的布局并将包括表格、标题、段落和图像在内的内容翻译成文档的文本表示。这个过程涉及到处理文本提取、图像识别中的不准确之处以及表中行-列关系的混乱。
二、如何解析PDF文档
一般来说解析PDF有三种方法
基于规则的方法根据文档的组织特征确定每个部分的风格和内容。然而这种方法不是很通用因为PDF有很多类型和布局不可能用预定义的规则覆盖所有类型和布局。基于深度学习模型的方法例如将目标检测和OCR模型相结合的流行解决方案。基于多模态大模型对复杂结构进行Pasing或提取PDF中的关键信息。
2.1 基于规则的方法 pypdf[1]就是一种基于规则广泛使用的解析器也是LangChain和LlamaIndex中解析PDF文件的标准方法。 以下是使用pypdf解析“Attention Is All You Need”[2]论文的第6页。原始页面如图3所示 代码如下
import PyPDF2filename /Users/Florian/Downloads/1706.03762.pdfpdf_file open(filename, rb)reader PyPDF2.PdfReader(pdf_file)page_num 5page reader.pages[page_num]text page.extract_text()print(--------------------------------------------------)print(text)pdf_file.close()
执行的结果是为了简洁起见省略了其余部分
(py) Florian:~ Florian$ pip list | grep pypdfpypdf 3.17.4pypdfium2 4.26.0(py) Florian:~ Florian$ python /Users/Florian/Downloads/pypdf_test.py--------------------------------------------------Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operationsfor different layer types. nis the sequence length, dis the representation dimension, kis the kernelsize of convolutions and rthe size of the neighborhood in restricted self-attention.Layer Type Complexity per Layer Sequential Maximum Path LengthOperationsSelf-Attention O(n2·d) O(1) O(1)Recurrent O(n·d2) O(n) O(n)Convolutional O(k·n·d2) O(1) O(logk(n))Self-Attention (restricted) O(r·n·d) O(1) O(n/r)3.5 Positional EncodingSince our model contains no recurrence and no convolution, in order for the model to make use of theorder of the sequence, we must inject some information about the relative or absolute position of thetokens in the sequence. To this end, we add positional encodings to the input embeddings at thebottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodelas the embeddings, so that the two can be summed. There are many choices of positional encodings,learned and fixed [9].In this work, we use sine and cosine functions of different frequencies:PE(pos,2i)sin(pos/100002i/d model)PE(pos,2i1)cos(pos/100002i/d model)where posis the position and iis the dimension. That is, each dimension of the positional encodingcorresponds to a sinusoid. The wavelengths form a geometric progression from 2πto10000 ·2π. Wechose this function because we hypothesized it would allow the model to easily learn to attend byrelative positions, since for any fixed offset k,PEposkcan be represented as a linear function ofPEpos.......... 从上述基于PyPDF检测的结果来看可以观察到它在不保留结构信息的情况下将PDF中的字符序列序列化为单个长序列。换句话说它将文档的每一行都视为一个由换行符“\n”分隔的序列这会妨碍段落或表格的准确识别。 这种限制是基于规则的方法的固有特征。
2.2 基于深度学习模型的方法 这种方法的优点是能够准确识别整个文档的布局包括表格和段落。它甚至可以理解表中的结构。这意味着它可以将文档划分为定义明确、完整的信息单元同时保留预期的含义和结构。 然而这种方法也有一些局限性目标检测和OCR阶段可能很耗时。因此建议使用GPU或其他加速设备并使用多个进程和线程进行处理。 这种方法涉及目标检测和OCR模型我测试了几个有代表性的开源框架
Unstructured[3]它已集成到langchain中[4]。使用hi_res策略设置infer_table_structureTrue可以很好的识别表格信息。然而fast策略因为不使用目标检测模型在识别图像和表格方面表现较差。Layout-parser[5]如果需要识别复杂的结构化PDF建议使用最大的模型以获得更高的精度尽管它可能会稍微慢一些。此外Layout解析器的模型[6]在过去两年中似乎没有更新。PP-StructureV2[7]可以组合各种模型用于文档分析性能高于平均水平。体系结构如图4所示 除了开源工具还有像ChatDOC这样的付费工具它们利用基于布局的识别OCR方法来解析PDF文档。 接下来我们将使用开源unstructured[3]解析PDF解决三个关键挑战。
挑战1如何从表格和图像中提取数据 在这里我们将使用unstructured[3]框架作为示例检测到的表数据可以直接导出为HTML。其代码如下
from unstructured.partition.pdf import partition_pdffilename /Users/Florian/Downloads/Attention_Is_All_You_Need.pdf# infer_table_structureTrue automatically selects hi_res strategyelements partition_pdf(filenamefilename, infer_table_structureTrue)tables [el for el in elements if el.category Table]print(tables[0].text)print(--------------------------------------------------)print(tables[0].metadata.text_as_html) partition_pdf函数的内部流程如下图5所示 代码的运行结果如下
Layer Type Self-Attention Recurrent Convolutional Self-Attention (restricted) Complexity per Layer O(n2 · d) O(n · d2) O(k · n · d2) O(r · n · d) Sequential Maximum Path Length Operations O(1) O(n) O(1) O(1) O(1) O(n) O(logk(n)) O(n/r)--------------------------------------------------tabletheadthLayer Type/ththComplexity per Layer/ththSequential Operations/ththMaximum Path Length/th/theadtrtdSelf-Attention/tdtdO(n? - d)/tdtdO(1)/tdtdO(1)/td/trtrtdRecurrent/tdtdO(n- d?)/tdtdO(n)/tdtdO(n)/td/trtrtdConvolutional/tdtdO(k-n-d?)/tdtdO(1)/tdtdO(logy(n))/td/trtrtdSelf-Attention (restricted)/tdtdO(r-n-d)/tdtdol)/tdtdO(n/r)/td/tr/table 复制HTML标记并将其另存为HTML文件。然后使用Chrome打开它如图6所示 可以观察到非结构化的算法在很大程度上恢复了整个表。
挑战2如何重新排列检测到的块特别是对于双列PDF 在处理双列PDF时让我们以论文“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”[8]为例读取顺序由红色箭头所示 在确定布局后unstructured[3]框架会将每个页面划分为几个矩形块如图8所示 每个矩形块的详细信息可以通过以下格式获得
[LayoutElement(bboxRectangle(x1851.1539916992188, y1181.15073777777613, x21467.844970703125, y2587.8204599999975), textThese approaches have been generalized to coarser granularities, such as sentence embed- dings (Kiros et al., 2015; Logeswaran and Lee, 2018) or paragraph embeddings (Le and Mikolov, 2014). To train sentence representations, prior work has used objectives to rank candidate next sentences (Jernite et al., 2017; Logeswaran and Lee, 2018), left-to-right generation of next sen- tence words given a representation of the previous sentence (Kiros et al., 2015), or denoising auto- encoder derived objectives (Hill et al., 2016). , sourceSource.YOLOX: yolox, typeText, prob0.9519357085227966, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1196.5296173095703, y1181.1507377777777, x2815.468994140625, y2512.548237777777), textword based only on its context. Unlike left-to- right language model pre-training, the MLM ob- jective enables the representation to fuse the left and the right context, which allows us to pre- In addi- train a deep bidirectional Transformer. tion to the masked language model, we also use a “next sentence prediction” task that jointly pre- trains text-pair representations. The contributions of our paper are as follows: , sourceSource.YOLOX: yolox, typeText, prob0.9517233967781067, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1200.22352600097656, y1539.1451822222216, x2825.0242919921875, y2870.542682222221), text• We demonstrate the importance of bidirectional pre-training for language representations. Un- like Radford et al. (2018), which uses unidirec- tional language models for pre-training, BERT uses masked language models to enable pre- trained deep bidirectional representations. This is also in contrast to Peters et al. (2018a), which uses a shallow concatenation of independently trained left-to-right and right-to-left LMs. , sourceSource.YOLOX: yolox, typeList-item, prob0.9414362907409668, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1851.8727416992188, y1599.8257377777753, x21468.0499267578125, y21420.4982377777742), textELMo and its predecessor (Peters et al., 2017, 2018a) generalize traditional word embedding re- search along a different dimension. They extract context-sensitive features from a left-to-right and a right-to-left language model. The contextual rep- resentation of each token is the concatenation of the left-to-right and right-to-left representations. When integrating contextual word embeddings with existing task-specific architectures, ELMo advances the state of the art for several major NLP benchmarks (Peters et al., 2018a) including ques- tion answering (Rajpurkar et al., 2016), sentiment analysis (Socher et al., 2013), and named entity recognition (Tjong Kim Sang and De Meulder, 2003). Melamud et al. (2016) proposed learning contextual representations through a task to pre- dict a single word from both left and right context using LSTMs. Similar to ELMo, their model is feature-based and not deeply bidirectional. Fedus et al. (2018) shows that the cloze task can be used to improve the robustness of text generation mod- els. , sourceSource.YOLOX: yolox, typeText, prob0.938507616519928, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1199.3734130859375, y1900.5257377777765, x2824.69873046875, y21156.648237777776), text• We show that pre-trained representations reduce the need for many heavily-engineered task- specific architectures. BERT is the first fine- tuning based representation model that achieves state-of-the-art performance on a large suite of sentence-level and token-level tasks, outper- forming many task-specific architectures. , sourceSource.YOLOX: yolox, typeList-item, prob0.9461237788200378, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1195.5695343017578, y11185.526123046875, x2815.9393920898438, y21330.3272705078125), text• BERT advances the state of the art for eleven NLP tasks. The code and pre-trained mod- els are available at https://github.com/ google-research/bert. , sourceSource.YOLOX: yolox, typeList-item, prob0.9213815927505493, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1195.33956909179688, y11360.7886962890625, x2447.47264000000007, y21397.038330078125), text2 Related Work , sourceSource.YOLOX: yolox, typeSection-header, prob0.8663332462310791, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1197.7477264404297, y11419.3353271484375, x2817.3308715820312, y21527.54443359375), textThere is a long history of pre-training general lan- guage representations, and we briefly review the most widely-used approaches in this section. , sourceSource.YOLOX: yolox, typeText, prob0.928022563457489, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1851.0028686523438, y11468.341394166663, x21420.4693603515625, y21498.6444497222187), text2.2 Unsupervised Fine-tuning Approaches , sourceSource.YOLOX: yolox, typeSection-header, prob0.8346447348594666, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1853.5444444444446, y11526.3701822222185, x21470.989990234375, y21669.5843488888852), textAs with the feature-based approaches, the first works in this direction only pre-trained word em- (Col- bedding parameters from unlabeled text lobert and Weston, 2008). , sourceSource.YOLOX: yolox, typeText, prob0.9344717860221863, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1200.00000000000009, y11556.2037353515625, x2799.1743774414062, y21588.031982421875), text2.1 Unsupervised Feature-based Approaches , sourceSource.YOLOX: yolox, typeSection-header, prob0.8317819237709045, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1198.64227294921875, y11606.3146266666645, x2815.2886352539062, y22125.895459999998), textLearning widely applicable representations of words has been an active area of research for decades, including non-neural (Brown et al., 1992; Ando and Zhang, 2005; Blitzer et al., 2006) and neural (Mikolov et al., 2013; Pennington et al., 2014) methods. Pre-trained word embeddings are an integral part of modern NLP systems, of- fering significant improvements over embeddings learned from scratch (Turian et al., 2010). To pre- train word embedding vectors, left-to-right lan- guage modeling objectives have been used (Mnih and Hinton, 2009), as well as objectives to dis- criminate correct from incorrect words in left and right context (Mikolov et al., 2013). , sourceSource.YOLOX: yolox, typeText, prob0.9450697302818298, image_pathNone, parentNone), LayoutElement(bboxRectangle(x1853.4905395507812, y11681.5868488888855, x21467.8729248046875, y22125.8954599999965), textMore recently, sentence or document encoders which produce contextual token representations have been pre-trained from unlabeled text and fine-tuned for a supervised downstream task (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018). The advantage of these approaches is that few parameters need to be learned from scratch. At least partly due to this advantage, OpenAI GPT (Radford et al., 2018) achieved pre- viously state-of-the-art results on many sentence- level tasks from the GLUE benchmark (Wang language model- Left-to-right et al., 2018a). , sourceSource.YOLOX: yolox, typeText, prob0.9476840496063232, image_pathNone, parentNone)] 其中x1y1是左上顶点的坐标x2y2是右下顶点的坐标 (x_1, y_1) -------- | | | | | | ---------- (x_2, y_2) 此时可以选择重新调整页面的阅读顺序。Unstructured[3]有一个内置的排序算法但我发现在处理双列情况时排序结果不是很令人满意。 因此有必要设计一种算法。最简单的方法是先按左上角顶点的水平坐标排序如果水平坐标相同则按垂直坐标排序。其伪代码如下所示
layout.sort(keylambda z: (z.bbox.x1, z.bbox.y1, z.bbox.x2, z.bbox.y2)) 然而我们发现即使是同一列中的块其水平坐标也可能发生变化。如图9所示紫色线条块的水平坐标bbox.x1实际上更靠左。排序时它将位于绿线块之前这显然违反了读取顺序。 在这种情况下使用的一种可能的算法如下
首先对左上角的所有x坐标x1进行排序我们可以得到x1_min然后对所有右下角的x坐标x2进行排序我们可以得到x2_max接下来将页面中心线的x坐标确定为
x1_min min([el.bbox.x1 for el in layout])x2_max max([el.bbox.x2 for el in layout])mid_line_x_coordinate (x2_max x1_min) / 2 接下来如果bbox.x1mid_line_x_cordinate则块被分类为左列的一部分。否则它将被视为右列的一部分。 分类完成后根据列中的y坐标对每个块进行排序。最后将右侧列连接到左侧列的右侧。
left_column []right_column []for el in layout: if el.bbox.x1 mid_line_x_coordinate: left_column.append(el) else: right_column.append(el)left_column.sort(key lambda z: z.bbox.y1)right_column.sort(key lambda z: z.bbox.y1)sorted_layout left_column right_column 值得一提的是这种改进也与单列PDF兼容。
挑战3如何提取多级标题 提取标题包括多级标题的目的是提高LLM答案的准确性。 例如如果用户想知道图9中2.1节的主要内容通过准确提取2.1节的标题并将其与相关内容一起作为上下文发送给LLM最终答案的准确性将显著提高。 该算法仍然依赖于图9所示的布局块。我们可以提取type’Section-header’的块并计算高度差bbox.y2--bbox.y1。高度差最大的块对应第一级标题其次是第二级标题然后是第三级标题。
2.3 基于多模态大模型解析复杂结构的PDF 在多模态模型爆炸之后也可以使用多模式模型来解析表。Llamalndex有几个例子[9]
检索相关图像PDF页面并将其发送到GPT4-V以响应查询。将每个PDF页面视为一个图像让GPT4-V对每个页面进行图像推理为图像推理构建文本矢量存储索引根据图像推理矢量存储查询答案。使用Table Transformer从检索到的图像中裁剪表信息然后将这些裁剪的图像发送到GPT4-V以进行查询响应。对裁剪的表图像应用OCR并将数据发送到GPT4/GGP-3.5以回答查询。 经过测试确定第三种方法是最有效的。 此外我们可以使用多模态模型从图像中提取或总结关键信息PDF文件可以很容易地转换为图像如图10所示 三、结论 一般来说非结构化文档提供了高度的灵活性并且需要各种解析技术。然而目前还没有达成共识的最佳实践。 在这种情况下建议选择最适合您项目需求的方法。建议根据不同类型的PDF应用特定的应对方法。例如论文、书籍和财务报表可能会根据其特点进行独特的设计。 然而如果可以的话建议选择基于深度学习或基于多模态的方法。这些方法可以有效地将文档分割成定义明确、完整的信息单元从而最大限度地保留文档的预期含义和结构。
参考文献
[1] https://github.com/py-pdf/pypdf
[2] https://arxiv.org/pdf/1706.03762.pdf
[3] http://unstructured-io.github.io/unstructured/
[4] https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/document_loaders/pdf.py
[5] http://github.com/Layout-Parser/layout-parser
[6] https://layout-parser.github.io/platform/
[7] https://arxiv.org/pdf/2210.05391.pdf
[8] https://arxiv.org/pdf/1810.04805.pdf
[9] https://docs.llamaindex.ai/en/stable/examples/multi_modal/multi_modal_pdf_tables.html
