wordpress网络,seo搜索引擎优化课程,济宁网络,学编程官网使用GraphRAG踩坑无数
在GraphRAG的使用过程中将需要踩的坑都踩了一遍#xff08;不得不吐槽下#xff0c;官方代码有很多遗留问题#xff0c;他们自己也承认工作重心在算法的优化而不是各种模型和框架的兼容性适配性上#xff09;#xff0c;经过了大量的查阅各种资料以…使用GraphRAG踩坑无数
在GraphRAG的使用过程中将需要踩的坑都踩了一遍不得不吐槽下官方代码有很多遗留问题他们自己也承认工作重心在算法的优化而不是各种模型和框架的兼容性适配性上经过了大量的查阅各种资料以及debug过程Indexing的过程有点费机器最终成功运行了GraphRAG项目。先后测试了两种方式都成功了:
使用ollama提供本地llm model和Embedding model服务使用ollama提供llm model服务使用lm-studio提供embedding model服务
之所以要使用ollama同时提供llm和Embedding模型服务是因为ollama实在是太优雅了使用超级简单响应速度也超级快。
使用ollama提供服务的方式如下
1、安装GraphRAG:
pip install graphrag -i https://pypi.tuna.tsinghua.edu.cn/simple创建一个文件路径:./ragtest/input
mkdir -p ./ragtest/input将语料文本文件放在这个路径下 文件格式为txt 注意txt文件必须是utf-8编码的可以用记事本打开另存为得到。使用命令python -m graphrag.index --init --root ./ragtest初始化工程:
python -m graphrag.index --init --root ./ragtest修改.env文件内容如下:
GRAPHRAG_API_KEYollama
GRAPHRAG_CLAIM_EXTRACTION_ENABLEDTrue注意必须加上参数GRAPHRAG_CLAIM_EXTRACTION_ENABLEDTrue否则无法生成协变量covariates 在Local Search时会出错。
修改.setting.yaml文件内容如下:
encoding_model: cl100k_base
skip_workflows: []
llm:api_key: ollamatype: openai_chat # or azure_openai_chatmodel: qwen2model_supports_json: true # recommended if this is available for your model.# max_tokens: 4000# request_timeout: 180.0api_base: http://localhost:11434/v1/# api_version: 2024-02-15-preview# organization: organization_id# deployment_name: azure_model_deployment_name# tokens_per_minute: 150_000 # set a leaky bucket throttle# requests_per_minute: 10_000 # set a leaky bucket throttle# max_retries: 10# max_retry_wait: 10.0# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times# concurrent_requests: 25 # the number of parallel inflight requests that may be madeparallelization:stagger: 0.3# num_threads: 50 # the number of threads to use for parallel processingasync_mode: threaded # or asyncioembeddings:## parallelization: override the global parallelization settings for embeddingsasync_mode: threaded # or asynciollm:api_key: ollamatype: openai_embedding # or azure_openai_embeddingmodel: nomic-embed-textapi_base: http://localhost:11434/v1/# api_version: 2024-02-15-preview# organization: organization_id# deployment_name: azure_model_deployment_name# tokens_per_minute: 150_000 # set a leaky bucket throttle# requests_per_minute: 10_000 # set a leaky bucket throttle# max_retries: 10# max_retry_wait: 10.0# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times# concurrent_requests: 25 # the number of parallel inflight requests that may be made# batch_size: 16 # the number of documents to send in a single request# batch_max_tokens: 8191 # the maximum number of tokens to send in a single request# target: required # or optional...
使用ollama启动llm和Embedding服务其中embedding 模型是nomic-embed-text:
ollama pull qwen2
ollama pull nomic-embed-text
ollama serve修改文件:D:\ProgramData\miniconda3\envs\graphRAG\Lib\site-packages\graphrag\llm\openai\openai_embeddings_llm.py内容(根据大家自己安装GraphRAG的路径查找)调用ollama服务:
import ollama# ....class OpenAIEmbeddingsLLM(BaseLLM[EmbeddingInput, EmbeddingOutput]):A text-embedding generator LLM._client: OpenAIClientTypes_configuration: OpenAIConfigurationdef __init__(self, client: OpenAIClientTypes, configuration: OpenAIConfiguration):self.client clientself.configuration configurationasync def _execute_llm(self, input: EmbeddingInput, **kwargs: Unpack[LLMInput]) - EmbeddingOutput | None:args {model: self.configuration.model,**(kwargs.get(model_parameters) or {}),}embedding await self.client.embeddings.create(inputinput,**args,)return [d.embedding for d in embedding.data]embedding_list []for inp in input:embedding ollama.embedding(modelnomic-embed-text,promptinp)embedding_list.append(embedding[embedding])return embedding_list
上面注释部分为官方原始代码增加的代码是: embedding_list []for inp in input:embedding ollama.embedding(modelnomic-embed-text,promptinp)embedding_list.append(embedding[embedding])return embedding_list修改文件D:\ProgramData\miniconda3\envs\graphRAG\Lib\site-packages\graphrag\query\llm\oai\embedding.py, 调用ollama提供的模型服务 代码位置在:
import ollama
#.....embedding ollama.embeddings(modelnomic-embed-text, promptchunk)[embedding]上面注释的是官方代码箭头指向的是要新增的代码。
修改文件:D:\ProgramData\miniconda3\envs\graphRAG\Lib\site-packages\graphrag\query\llm\text_utils.py里关于chunk_text()函数的定义:
def chunk_text(text: str, max_tokens: int, token_encoder: tiktoken.Encoding | None None
):Chunk text by token length.if token_encoder is None:token_encoder tiktoken.get_encoding(cl100k_base)tokens token_encoder.encode(text) # type: ignoretokens token_encoder.decode(tokens) # 将tokens解码成字符串chunk_iterator batched(iter(tokens), max_tokens)yield from chunk_iterator增加的语句是:
tokens token_encoder.decode(tokens) # 将tokens解码成字符串这里应该是GraphRAG官方代码里的bug开发人员忘记将分词后的token解码成字符串导致在后续Embedding处理过程中会报错ZeroDivisionError: Weights sum to zero, cant be normalized
(graphrag) D:\Learn\GraphRAGpython -m graphrag.query --root ./newTest12 --method local 谁是叶文洁INFO: Reading settings from newTest12\settings.yaml
creating llm client with {api_key: REDACTED,len6, type: openai_chat, model: qwen2, max_tokens: 4000, temperature: 0.0, top_p: 1.0, n: 1, request_timeout: 180.0, api_base: http://localhost:11434/v1/, api_version: None, organization: None, proxy: None, cognitive_services_endpoint: None, deployment_name: None, model_supports_json: True, tokens_per_minute: 0, requests_per_minute: 0, max_retries: 10, max_retry_wait: 10.0, sleep_on_rate_limit_recommendation: True, concurrent_requests: 25}
creating embedding llm client with {api_key: REDACTED,len9, type: openai_embedding, model: nomic-ai/nomic-embed-text-v1.5/nomic-embed-text-v1.5.Q8_0.gguf, max_tokens: 4000, temperature: 0, top_p: 1, n: 1, request_timeout: 180.0, api_base: http://localhost:1234/v1, api_version: None, organization: None, proxy: None, cognitive_services_endpoint: None, deployment_name: None, model_supports_json: None, tokens_per_minute: 0, requests_per_minute: 0, max_retries: 10, max_retry_wait: 10.0, sleep_on_rate_limit_recommendation: True, concurrent_requests: 1}
Error embedding chunk {OpenAIEmbedding: Error code: 400 - {\error\: \input\ field must be a string or an array of strings}}
Traceback (most recent call last):File D:\ProgramData\miniconda3\envs\graphrag\lib\runpy.py, line 196, in _run_module_as_mainreturn _run_code(code, main_globals, None,File D:\ProgramData\miniconda3\envs\graphrag\lib\runpy.py, line 86, in _run_codeexec(code, run_globals)File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\__main__.py, line 76, in modulerun_local_search(File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\cli.py, line 153, in run_local_searchresult search_engine.search(queryquery)File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\structured_search\local_search\search.py, line 118, in searchcontext_text, context_records self.context_builder.build_context(File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\structured_search\local_search\mixed_context.py, line 139, in build_contextselected_entities map_query_to_entities(File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\context_builder\entity_extraction.py, line 55, in map_query_to_entitiessearch_results text_embedding_vectorstore.similarity_search_by_text(File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\vector_stores\lancedb.py, line 118, in similarity_search_by_textquery_embedding text_embedder(text)File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\context_builder\entity_extraction.py, line 57, in lambdatext_embedderlambda t: text_embedder.embed(t),File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\graphrag\query\llm\oai\embedding.py, line 96, in embedchunk_embeddings np.average(chunk_embeddings, axis0, weightschunk_lens)File D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\lib\function_base.py, line 550, in averageraise ZeroDivisionError(
ZeroDivisionError: Weights sum to zero, cant be normalized开始Indexing处理:
python -m graphrag.index --root ./ragtest运行效果:
(graphrag) D:\Learn\GraphRAGpython -m graphrag.index --root ./newTest12Reading settings from newTest12\settings.yaml
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)create_base_text_unitsid ... n_tokens
0 eb94998b0499b6271136701074a1d890 ... 300
1 ae83a5ece6993bb8441110c128374267 ... 300
2 8debc287482f854d941a17262b4fe9b4 ... 300
3 0afae36282bd8db18b85ed0ff5c6bfcf ... 300
4 6029ac47ac05acb22ae6b625c2e726e5 ... 300
5 18a2202cc4756368e833007edc118b83 ... 300
6 0f1ca0e967c49c0eccb0641e4dca1d07 ... 300
7 0f2c27b592f5ed732eb5dbf041475950 ... 300
8 319702df76e338acb4ad3d0e02dd3d6f ... 300
9 919746c8d00d55401129a3eb6eb335d9 ... 300
10 4cf72e5c48316b181b279c62ada7ee6d ... 300
11 6a7c6d9db387332aa7d9178d22014fa6 ... 300
12 bd7e44fb9063cf8e02da39443f4c67eb ... 300
13 3239241f8fba889b9ebd1851c4f68aa5 ... 300
14 c9d05edb3d1a58711f42639e18cdcea2 ... 300
15 a4c53469e9283bad549f1d10568bba4b ... 300
16 01e50959b91fc167df1bd0fe83f2928b ... 300
17 91d7b0359c7417bd8c4ff0931c6ba236 ... 300
18 0c2f21e8f141de2a2e03f17a875de54a ... 300
19 7716c29d83922f69e228eca2c99128ce ... 300
20 af2ef2f39176a565b509d48ef91f5ca6 ... 300
21 38a919532f499e6c873162a050619f31 ... 300
22 587fbda555a7a3a371ae35b16084f555 ... 300
23 4dbcb435fc91cdbe2bbd4ca075e7df4d ... 300
24 a08a77fbbf1ea343ef915b776beb4fad ... 300
25 5d57d8d015e8d98ef355f0f42e114bb0 ... 300
26 cba7a1ca9b4099be67035d5263d3cbab ... 300
27 403ee5e0425c850acea5f66494ab5590 ... 300
28 f19574bd0b5f9db26188fbe7ce063035 ... 300
29 f0577fe53579d7da7f4bded3cc209220 ... 300
30 01ba18a8dc1159200e6e5418392b2de1 ... 300
31 3bec09f620a572b869885b19b82c520e ... 300
32 8081e9512c0bd1163378659ea18fa589 ... 300
33 78fb8731a8b51236488c07546bb39ab0 ... 300
34 949ee97d8a055ea639b65db190326580 ... 300
35 d7c149cd8df10e29d99c0a257cbab60f ... 300
36 42241043af1a3ae708fe06d4644b79fe ... 300
37 824ff7fe74b00fa6af083d9c42bfe0ef ... 300
38 43adb8cbfbfb7f8631ff19988d27f8f0 ... 300
39 a621a38808af24546ac397393e8bc6be ... 300
40 5ee1a053b42c395db7c0abdc55e88af7 ... 300
41 364150258ec05bb31b80141b75d7a5ca ... 300
42 d760b8e30ecd977add71ba4274b0c9dd ... 300
43 ebf935b232b056a6973cb6763a532a43 ... 300
44 299966570cf5d14d7d46a4a81555907b ... 300
45 d6e4272bf5306dd8d1054e9a56ad7114 ... 200[46 rows x 5 columns]create_base_extracted_entitiesentity_graph
0 graphml xmlnshttp://graphml.graphdrawing.or...
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\datashaper\engine\verbs\convert.py:65: FutureWarning:
errorsignore is deprecated and will raise in a future version. Use to_numeric without passing errors and catch
exceptions explicitly insteadcolumn_numeric cast(pd.Series, pd.to_numeric(column, errorsignore))create_final_covariatesid human_readable_id ... document_ids n_tokens
0 fa863911-f68e-4f11-bf1f-5c074ce528c8 1 ... [9907241b0721ab0f48fbbc9d784175eb] 300
1 6245da46-086e-476c-b4b7-b3efc1bd82bb 2 ... [9907241b0721ab0f48fbbc9d784175eb] 300
2 7f4ee402-0065-4b2e-a5d8-3eef944b18f3 3 ... [9907241b0721ab0f48fbbc9d784175eb] 300
3 1927e65b-3a8c-4c3a-bda8-4bbc1804737f 4 ... [9907241b0721ab0f48fbbc9d784175eb] 300
4 ebb53a51-9f03-4ede-924b-93f6f74320da 5 ... [9907241b0721ab0f48fbbc9d784175eb] 300
.. ... ... ... ... ...
56 81dc46bc-1c00-46a8-b745-aae710bfd949 57 ... [9907241b0721ab0f48fbbc9d784175eb] 300
57 c96c929d-80be-4fc5-a865-ced074fe2f01 58 ... [9907241b0721ab0f48fbbc9d784175eb] 300
58 785b12a8-3669-48fc-a017-f8fa1b60348e 59 ... [9907241b0721ab0f48fbbc9d784175eb] 300
59 47cb429c-c402-4eb9-bcab-4c427cea6176 60 ... [9907241b0721ab0f48fbbc9d784175eb] 300
60 701529fc-1499-4efe-bdac-0bc3a49a942c 61 ... [9907241b0721ab0f48fbbc9d784175eb] 200[61 rows x 16 columns]create_summarized_entitiesentity_graph
0 graphml xmlnshttp://graphml.graphdrawing.or...join_text_units_to_covariate_idstext_unit_id ... id
0 eb94998b0499b6271136701074a1d890 ... eb94998b0499b6271136701074a1d890
1 ae83a5ece6993bb8441110c128374267 ... ae83a5ece6993bb8441110c128374267
2 8debc287482f854d941a17262b4fe9b4 ... 8debc287482f854d941a17262b4fe9b4
3 0afae36282bd8db18b85ed0ff5c6bfcf ... 0afae36282bd8db18b85ed0ff5c6bfcf
4 6029ac47ac05acb22ae6b625c2e726e5 ... 6029ac47ac05acb22ae6b625c2e726e5
5 18a2202cc4756368e833007edc118b83 ... 18a2202cc4756368e833007edc118b83
6 0f1ca0e967c49c0eccb0641e4dca1d07 ... 0f1ca0e967c49c0eccb0641e4dca1d07
7 0f2c27b592f5ed732eb5dbf041475950 ... 0f2c27b592f5ed732eb5dbf041475950
8 319702df76e338acb4ad3d0e02dd3d6f ... 319702df76e338acb4ad3d0e02dd3d6f
9 919746c8d00d55401129a3eb6eb335d9 ... 919746c8d00d55401129a3eb6eb335d9
10 4cf72e5c48316b181b279c62ada7ee6d ... 4cf72e5c48316b181b279c62ada7ee6d
11 6a7c6d9db387332aa7d9178d22014fa6 ... 6a7c6d9db387332aa7d9178d22014fa6
12 bd7e44fb9063cf8e02da39443f4c67eb ... bd7e44fb9063cf8e02da39443f4c67eb
13 3239241f8fba889b9ebd1851c4f68aa5 ... 3239241f8fba889b9ebd1851c4f68aa5
14 c9d05edb3d1a58711f42639e18cdcea2 ... c9d05edb3d1a58711f42639e18cdcea2
15 a4c53469e9283bad549f1d10568bba4b ... a4c53469e9283bad549f1d10568bba4b
16 01e50959b91fc167df1bd0fe83f2928b ... 01e50959b91fc167df1bd0fe83f2928b
17 91d7b0359c7417bd8c4ff0931c6ba236 ... 91d7b0359c7417bd8c4ff0931c6ba236
18 0c2f21e8f141de2a2e03f17a875de54a ... 0c2f21e8f141de2a2e03f17a875de54a
19 7716c29d83922f69e228eca2c99128ce ... 7716c29d83922f69e228eca2c99128ce
20 af2ef2f39176a565b509d48ef91f5ca6 ... af2ef2f39176a565b509d48ef91f5ca6
21 38a919532f499e6c873162a050619f31 ... 38a919532f499e6c873162a050619f31
22 587fbda555a7a3a371ae35b16084f555 ... 587fbda555a7a3a371ae35b16084f555
23 4dbcb435fc91cdbe2bbd4ca075e7df4d ... 4dbcb435fc91cdbe2bbd4ca075e7df4d
24 a08a77fbbf1ea343ef915b776beb4fad ... a08a77fbbf1ea343ef915b776beb4fad
25 5d57d8d015e8d98ef355f0f42e114bb0 ... 5d57d8d015e8d98ef355f0f42e114bb0
26 f0577fe53579d7da7f4bded3cc209220 ... f0577fe53579d7da7f4bded3cc209220
27 8081e9512c0bd1163378659ea18fa589 ... 8081e9512c0bd1163378659ea18fa589
28 78fb8731a8b51236488c07546bb39ab0 ... 78fb8731a8b51236488c07546bb39ab0
29 949ee97d8a055ea639b65db190326580 ... 949ee97d8a055ea639b65db190326580
30 d7c149cd8df10e29d99c0a257cbab60f ... d7c149cd8df10e29d99c0a257cbab60f
31 42241043af1a3ae708fe06d4644b79fe ... 42241043af1a3ae708fe06d4644b79fe
32 824ff7fe74b00fa6af083d9c42bfe0ef ... 824ff7fe74b00fa6af083d9c42bfe0ef
33 a621a38808af24546ac397393e8bc6be ... a621a38808af24546ac397393e8bc6be
34 ebf935b232b056a6973cb6763a532a43 ... ebf935b232b056a6973cb6763a532a43
35 299966570cf5d14d7d46a4a81555907b ... 299966570cf5d14d7d46a4a81555907b
36 d6e4272bf5306dd8d1054e9a56ad7114 ... d6e4272bf5306dd8d1054e9a56ad7114[37 rows x 3 columns]create_base_entity_graphlevel clustered_graph
0 0 graphml xmlnshttp://graphml.graphdrawing.or...
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)create_final_entitiesid ... description_embedding
0 b45241d70f0e43fca764df95b2b81f77 ... [-0.037392858415842056, 0.06525952368974686, -...
1 4119fd06010c494caa07f439b333f4c5 ... [0.010907179675996304, 0.026875361800193787, -...
2 d3835bf3dda84ead99deadbeac5d0d7d ... [0.054428134113550186, -9.018656419357285e-05,...
3 077d2820ae1845bcbb1803379a3d1eae ... [0.020732643082737923, 0.0034371891524642706, ...
4 3671ea0dd4e84c1a9b02c5ab2c8f4bac ... [-0.0012893152888864279, 0.037432845681905746,...
.. ... ... ...
59 958beecdb5bb4060948415ffd75d2b03 ... [0.01642344333231449, 0.021773478016257286, -0...
60 b999ed77e19e4f85b7f1ae79af5c002a ... [0.002400514902547002, 0.047308988869190216, -...
61 48c0c4d72da74ff5bb926fa0c856d1a7 ... [-0.01692129857838154, 0.0539858303964138, -0....
62 4f3c97517f794ebfb49c4c6315f9cf23 ... [0.0010956701589748263, 0.04648151248693466, -...
63 1745a2485a9443bab76587ad650e9be0 ... [-0.007561820093542337, 0.045520562678575516, ...[64 rows x 8 columns]
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\datashaper\engine\verbs\convert.py:72: FutureWarning:
errorsignore is deprecated and will raise in a future version. Use to_datetime without passing errors and catch
exceptions explicitly insteaddatetime_column pd.to_datetime(column, errorsignore)
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\datashaper\engine\verbs\convert.py:72: UserWarning: Could not
infer format, so each element will be parsed individually, falling back to dateutil. To ensure parsing is consistent
and as-expected, please specify a format.datetime_column pd.to_datetime(column, errorsignore)create_final_nodeslevel title type ... top_level_node_id x y
0 0 红色联合 ORGANIZATION ... b45241d70f0e43fca764df95b2b81f77 0 0
1 0 四·二八兵团 ORGANIZATION ... 4119fd06010c494caa07f439b333f4c5 0 0
2 0 1967年 EVENT ... d3835bf3dda84ead99deadbeac5d0d7d 0 0
3 0 安眠药瓶 OBJECT ... 077d2820ae1845bcbb1803379a3d1eae 0 0
4 0 铁炉子 GEO ... 3671ea0dd4e84c1a9b02c5ab2c8f4bac 0 0
.. ... ... ... ... ... .. ..
59 0 老校工 ORGANIZATION ... 958beecdb5bb4060948415ffd75d2b03 0 0
60 0 教工宿舍楼 GEO ... b999ed77e19e4f85b7f1ae79af5c002a 0 0
61 0 阮老师 PERSON ... 48c0c4d72da74ff5bb926fa0c856d1a7 0 0
62 0 阮雯 PERSON ... 4f3c97517f794ebfb49c4c6315f9cf23 0 0
63 0 文洁 PERSON ... 1745a2485a9443bab76587ad650e9be0 0 0[64 rows x 14 columns]
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)create_final_communitiesid title ... relationship_ids
text_unit_ids
0 0 Community 0 ... [32e6ccab20d94029811127dbbe424c64, 94a964c6992...
[0f2c27b592f5ed732eb5dbf041475950,a621a38808af...[1 rows x 6 columns]join_text_units_to_entity_idstext_unit_ids ... id
0 0f1ca0e967c49c0eccb0641e4dca1d07 ... 0f1ca0e967c49c0eccb0641e4dca1d07
1 18a2202cc4756368e833007edc118b83 ... 18a2202cc4756368e833007edc118b83
2 6029ac47ac05acb22ae6b625c2e726e5 ... 6029ac47ac05acb22ae6b625c2e726e5
3 eb94998b0499b6271136701074a1d890 ... eb94998b0499b6271136701074a1d890
4 0c2f21e8f141de2a2e03f17a875de54a ... 0c2f21e8f141de2a2e03f17a875de54a
5 0f2c27b592f5ed732eb5dbf041475950 ... 0f2c27b592f5ed732eb5dbf041475950
6 319702df76e338acb4ad3d0e02dd3d6f ... 319702df76e338acb4ad3d0e02dd3d6f
7 3bec09f620a572b869885b19b82c520e ... 3bec09f620a572b869885b19b82c520e
8 403ee5e0425c850acea5f66494ab5590 ... 403ee5e0425c850acea5f66494ab5590
9 5d57d8d015e8d98ef355f0f42e114bb0 ... 5d57d8d015e8d98ef355f0f42e114bb0
10 5ee1a053b42c395db7c0abdc55e88af7 ... 5ee1a053b42c395db7c0abdc55e88af7
11 949ee97d8a055ea639b65db190326580 ... 949ee97d8a055ea639b65db190326580
12 af2ef2f39176a565b509d48ef91f5ca6 ... af2ef2f39176a565b509d48ef91f5ca6
13 f19574bd0b5f9db26188fbe7ce063035 ... f19574bd0b5f9db26188fbe7ce063035
14 ae83a5ece6993bb8441110c128374267 ... ae83a5ece6993bb8441110c128374267
15 8debc287482f854d941a17262b4fe9b4 ... 8debc287482f854d941a17262b4fe9b4
16 0afae36282bd8db18b85ed0ff5c6bfcf ... 0afae36282bd8db18b85ed0ff5c6bfcf
17 3239241f8fba889b9ebd1851c4f68aa5 ... 3239241f8fba889b9ebd1851c4f68aa5
18 c9d05edb3d1a58711f42639e18cdcea2 ... c9d05edb3d1a58711f42639e18cdcea2
19 a621a38808af24546ac397393e8bc6be ... a621a38808af24546ac397393e8bc6be
20 919746c8d00d55401129a3eb6eb335d9 ... 919746c8d00d55401129a3eb6eb335d9
21 4cf72e5c48316b181b279c62ada7ee6d ... 4cf72e5c48316b181b279c62ada7ee6d
22 6a7c6d9db387332aa7d9178d22014fa6 ... 6a7c6d9db387332aa7d9178d22014fa6
23 8081e9512c0bd1163378659ea18fa589 ... 8081e9512c0bd1163378659ea18fa589
24 91d7b0359c7417bd8c4ff0931c6ba236 ... 91d7b0359c7417bd8c4ff0931c6ba236
25 a4c53469e9283bad549f1d10568bba4b ... a4c53469e9283bad549f1d10568bba4b
26 bd7e44fb9063cf8e02da39443f4c67eb ... bd7e44fb9063cf8e02da39443f4c67eb
27 d7c149cd8df10e29d99c0a257cbab60f ... d7c149cd8df10e29d99c0a257cbab60f
28 f0577fe53579d7da7f4bded3cc209220 ... f0577fe53579d7da7f4bded3cc209220
29 01e50959b91fc167df1bd0fe83f2928b ... 01e50959b91fc167df1bd0fe83f2928b
30 7716c29d83922f69e228eca2c99128ce ... 7716c29d83922f69e228eca2c99128ce
31 38a919532f499e6c873162a050619f31 ... 38a919532f499e6c873162a050619f31
32 587fbda555a7a3a371ae35b16084f555 ... 587fbda555a7a3a371ae35b16084f555
33 4dbcb435fc91cdbe2bbd4ca075e7df4d ... 4dbcb435fc91cdbe2bbd4ca075e7df4d
34 a08a77fbbf1ea343ef915b776beb4fad ... a08a77fbbf1ea343ef915b776beb4fad
35 cba7a1ca9b4099be67035d5263d3cbab ... cba7a1ca9b4099be67035d5263d3cbab
36 01ba18a8dc1159200e6e5418392b2de1 ... 01ba18a8dc1159200e6e5418392b2de1
37 78fb8731a8b51236488c07546bb39ab0 ... 78fb8731a8b51236488c07546bb39ab0
38 42241043af1a3ae708fe06d4644b79fe ... 42241043af1a3ae708fe06d4644b79fe
39 824ff7fe74b00fa6af083d9c42bfe0ef ... 824ff7fe74b00fa6af083d9c42bfe0ef
40 43adb8cbfbfb7f8631ff19988d27f8f0 ... 43adb8cbfbfb7f8631ff19988d27f8f0
41 299966570cf5d14d7d46a4a81555907b ... 299966570cf5d14d7d46a4a81555907b
42 364150258ec05bb31b80141b75d7a5ca ... 364150258ec05bb31b80141b75d7a5ca
43 d760b8e30ecd977add71ba4274b0c9dd ... d760b8e30ecd977add71ba4274b0c9dd
44 ebf935b232b056a6973cb6763a532a43 ... ebf935b232b056a6973cb6763a532a43
45 d6e4272bf5306dd8d1054e9a56ad7114 ... d6e4272bf5306dd8d1054e9a56ad7114[46 rows x 3 columns]
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\numpy\core\fromnumeric.py:59: FutureWarning:
DataFrame.swapaxes is deprecated and will be removed in a future version. Please use DataFrame.transpose instead.return bound(*args, **kwds)
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\datashaper\engine\verbs\convert.py:65: FutureWarning:
errorsignore is deprecated and will raise in a future version. Use to_numeric without passing errors and catch
exceptions explicitly insteadcolumn_numeric cast(pd.Series, pd.to_numeric(column, errorsignore))create_final_relationshipssource target weight ... source_degree target_degree rank
0 SHE CULTURAL REVOLUTION 1.0 ... 1 2 3
1 THE CITY CULTURAL REVOLUTION 1.0 ... 1 2 3
2 RED GUARDS FEMALE RED GUARD 1.0 ... 2 1 3
3 RED GUARDS MALE RED GUARD 1.0 ... 2 1 3
4 小红卫兵 QUESTIONING 1.0 ... 1 1 2[5 rows x 10 columns]join_text_units_to_relationship_idsid relationship_ids
0 0f2c27b592f5ed732eb5dbf041475950 [32e6ccab20d94029811127dbbe424c64, 94a964c6992...
1 cba7a1ca9b4099be67035d5263d3cbab [1eb829d0ace042089f0746f78729696c, 015e7b58d1a...
2 8081e9512c0bd1163378659ea18fa589 [26f88ab3e2e04c33a459ad6270ade565]create_final_community_reportscommunity ... id
0 0 ... a1ceb1f1-c824-420b-a93f-2a76e83a4398[1 rows x 10 columns]create_final_text_unitsid ... covariate_ids
0 0f2c27b592f5ed732eb5dbf041475950 ... [2c940a06-373b-402e-9203-b7b43b5ff0a4, d5dcbf1...
1 8081e9512c0bd1163378659ea18fa589 ... [4a8f80d6-6509-4470-a6f2-788fbe81f52e, cbc8bf5...
2 eb94998b0499b6271136701074a1d890 ... [fa863911-f68e-4f11-bf1f-5c074ce528c8, 6245da4...
3 ae83a5ece6993bb8441110c128374267 ... [1927e65b-3a8c-4c3a-bda8-4bbc1804737f]
4 8debc287482f854d941a17262b4fe9b4 ...
5 0afae36282bd8db18b85ed0ff5c6bfcf ... [5e0d4564-20f6-4d9e-b562-7ffe3f44278d, 2069b5d...
6 6029ac47ac05acb22ae6b625c2e726e5 ... [f3a2ef27-fb45-473a-bbc9-e43cb9d34d1c, 63dd709...
7 18a2202cc4756368e833007edc118b83 ... [0eb5023a-8012-4881-8593-2de54301c8bb]
8 0f1ca0e967c49c0eccb0641e4dca1d07 ...
9 319702df76e338acb4ad3d0e02dd3d6f ... [423c8608-0d59-41f2-9197-ae612f1239e0]
10 919746c8d00d55401129a3eb6eb335d9 ... [3d7ecd82-20ac-438e-ac86-997f6ad58cc5]
11 4cf72e5c48316b181b279c62ada7ee6d ... [82df9600-0bb7-4d0b-950c-067740692784, f89ecbc...
12 6a7c6d9db387332aa7d9178d22014fa6 ... [1d4aff9a-f347-4aea-b255-8b9c092421c4]
13 bd7e44fb9063cf8e02da39443f4c67eb ...
14 3239241f8fba889b9ebd1851c4f68aa5 ... [87efcce8-fbfb-4806-b2ce-834b2a7327c9, aecb7f3...
15 c9d05edb3d1a58711f42639e18cdcea2 ... [467b2889-ad04-4d39-b84f-d0567fe220ce]
16 a4c53469e9283bad549f1d10568bba4b ...
17 01e50959b91fc167df1bd0fe83f2928b ... [8afcb698-9fb9-4ee9-bb38-49c854f1f9b6]
18 91d7b0359c7417bd8c4ff0931c6ba236 ...
19 0c2f21e8f141de2a2e03f17a875de54a ... [68e16f47-8b94-4f3f-bf8f-30042b0d797e]
20 7716c29d83922f69e228eca2c99128ce ... [bf2f48b8-3f39-453c-a020-b8e3c4937f43, f2593f9...
21 af2ef2f39176a565b509d48ef91f5ca6 ... [9f6423b6-3168-4650-bc27-e7f7d3b4eee1]
22 38a919532f499e6c873162a050619f31 ... [5a98452e-b66f-4ba8-995a-384a9907424a]
23 587fbda555a7a3a371ae35b16084f555 ... [628e6f7c-b9ef-494f-a2b6-c5e9ffe58fab, 9943f75...
24 4dbcb435fc91cdbe2bbd4ca075e7df4d ... [141cdefd-3e39-41d3-9a05-7b4d3a0e3cda]
25 a08a77fbbf1ea343ef915b776beb4fad ... [11d29b8f-f528-455a-af29-0af3dd9c1f69]
26 5d57d8d015e8d98ef355f0f42e114bb0 ... [b26c0619-4051-4b31-80bb-ba064c7153bd, c12d27f...
27 f0577fe53579d7da7f4bded3cc209220 ... [1b5269e5-7cdd-4485-ae8d-ed7dffaadda4]
28 78fb8731a8b51236488c07546bb39ab0 ...
29 949ee97d8a055ea639b65db190326580 ... [97e3724c-eca5-43ed-a308-f23296458464]
30 d7c149cd8df10e29d99c0a257cbab60f ... [94eb196e-5a69-4dd4-87dd-92746a88215c]
31 42241043af1a3ae708fe06d4644b79fe ... [56c81b53-0bfd-44e3-98dd-3b69d4997b68]
32 824ff7fe74b00fa6af083d9c42bfe0ef ... [81dc46bc-1c00-46a8-b745-aae710bfd949]
33 a621a38808af24546ac397393e8bc6be ...
34 ebf935b232b056a6973cb6763a532a43 ... [785b12a8-3669-48fc-a017-f8fa1b60348e]
35 299966570cf5d14d7d46a4a81555907b ... [47cb429c-c402-4eb9-bcab-4c427cea6176]
36 d6e4272bf5306dd8d1054e9a56ad7114 ... [701529fc-1499-4efe-bdac-0bc3a49a942c]
37 cba7a1ca9b4099be67035d5263d3cbab ... None
38 403ee5e0425c850acea5f66494ab5590 ... None
39 f19574bd0b5f9db26188fbe7ce063035 ... None
40 01ba18a8dc1159200e6e5418392b2de1 ... None
41 3bec09f620a572b869885b19b82c520e ... None
42 43adb8cbfbfb7f8631ff19988d27f8f0 ... None
43 5ee1a053b42c395db7c0abdc55e88af7 ... None
44 364150258ec05bb31b80141b75d7a5ca ... None
45 d760b8e30ecd977add71ba4274b0c9dd ... None[46 rows x 7 columns]
D:\ProgramData\miniconda3\envs\graphrag\lib\site-packages\datashaper\engine\verbs\convert.py:72: FutureWarning:
errorsignore is deprecated and will raise in a future version. Use to_datetime without passing errors and catch
exceptions explicitly insteaddatetime_column pd.to_datetime(column, errorsignore)create_base_documentsid ... title
0 9907241b0721ab0f48fbbc9d784175eb ... 01.txt[1 rows x 4 columns]create_final_documentsid ... title
0 9907241b0721ab0f48fbbc9d784175eb ... 01.txt[1 rows x 4 columns]
⠏ GraphRAG Indexer
├── Loading Input (text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00
├── create_base_text_units
├── create_base_extracted_entities
├── create_final_covariates
├── create_summarized_entities
├── join_text_units_to_covariate_ids
├── create_base_entity_graph
├── create_final_entities
├── create_final_nodes
├── create_final_communities
├── join_text_units_to_entity_ids
├── create_final_relationships
├── join_text_units_to_relationship_ids
├── create_final_community_reports
├── create_final_text_units
├── create_base_documents
└── create_final_documentsAll workflows completed successfully.12 . 执行全局查询global Search:
python -m graphrag.query --root ./newTest12 --method global 谁是叶文洁运行效果:
(graphrag) D:\Learn\GraphRAGpython -m graphrag.query --root ./newTest12 --method global 谁是叶文洁INFO: Reading settings from newTest12\settings.yaml
creating llm client with {api_key: REDACTED,len6, type: openai_chat, model: qwen2, max_tokens: 4000, temperature: 0.0, top_p: 1.0, n: 1, request_timeout: 180.0, api_base: http://localhost:11434/v1/, api_version: None, organization: None, proxy: None, cognitive_services_endpoint: None, deployment_name: None, model_supports_json: True, tokens_per_minute: 0, requests_per_minute: 0, max_retries: 10, max_retry_wait: 10.0, sleep_on_rate_limit_recommendation: True, concurrent_requests: 25}SUCCESS: Global Search Response: 叶文洁是一位在《三体》系列小说中扮演重要角色的科学家。她是中国第一位天线物理学家在故事早期阶段对研究三体文明做出了贡献。根据分析师1的报告叶文洁的身份和背景在《三体》系列中被详细描绘。她是该系列中的关键人物之一通过她的科学工作和对三体文明的研究为整个故事的发展提供了重要的推动力。因此我们可以得出结论叶文洁是一位在科幻小说《三体》系列中具有重要地位的科学家角色。请注意分析师报告中提到的具体数据记录如编号2、7、34、46、64等用于支持上述信息但为了简洁起见在此未详细列出。这些数据记录提供了关于叶文洁在小说中的具体描述和背景信息。执行局部查询Local search:
python -m graphrag.query --root ./newTest12 --method local 谁是叶文洁运行效果:
SUCCESS: Local Search Response: 叶文洁是中国科幻小说《三体》系列中的一个主要角色由刘慈欣所创造。在故事中她是一位天体物理学家和工程师在中国科学院工作并参与了“红岸工程”这是中国的一个外星文明探测项目。 叶文洁因为对人类社会的失望以及对宇宙探索的热情而选择与外星文明接触这一行为导致了她的职业生涯遭受重创。在《三体》系列中叶文洁的故事线贯穿整个故事她经历了从科学家到被追捕者、再到成为抵抗组织核心成员的角色转变。她对于人类社会的失望和对未知宇宙的好奇心使得她在面对外星文明时有着独特的视角和行动方式。叶文洁的 形象在科幻文学中具有一定的代表性展现了人性中的复杂性和对未知世界探索的渴望。《三体》系列是中国科幻文学的重要作品之一获得了包括“雨果奖”在内的多个奖项深受读者喜爱并在全球范围内产生了广泛影响。查看大模型回答问题所依赖的上下文这时需要使用GraphRAG 的python调用方式:
import osimport pandas as pd
import tiktoken # Tiktoken 是一种文本处理工具它能够将文本分解成更小的单元通常用于自然语言处理NLP任务中的文本编码。from graphrag.query.context_builder.entity_extraction import EntityVectorStoreKey
from graphrag.query.indexer_adapters import (read_indexer_covariates,read_indexer_entities,read_indexer_relationships,read_indexer_reports,read_indexer_text_units,
)
from graphrag.query.input.loaders.dfs import (store_entity_semantic_embeddings,
)
from graphrag.query.llm.oai.chat_openai import ChatOpenAI
from graphrag.query.llm.oai.embedding import OpenAIEmbedding
from graphrag.query.llm.oai.typing import OpenaiApiType
from graphrag.query.question_gen.local_gen import LocalQuestionGen
from graphrag.query.structured_search.local_search.mixed_context import (LocalSearchMixedContext,
)
from graphrag.query.structured_search.local_search.search import LocalSearch
from graphrag.vector_stores.lancedb import LanceDBVectorStore# 配置参数
INPUT_DIR ../newTest12/output/20240802-103645/artifacts # 这里换成所在工程的输出路径
LANCEDB_URI f./lancedbCOMMUNITY_REPORT_TABLE create_final_community_reports
ENTITY_TABLE create_final_nodes
ENTITY_EMBEDDING_TABLE create_final_entities
RELATIONSHIP_TABLE create_final_relationships
COVARIATE_TABLE create_final_covariates
TEXT_UNIT_TABLE create_final_text_units
COMMUNITY_LEVEL 2# 读取实体entities
# read nodes table to get community and degree data
entity_df pd.read_parquet(f{INPUT_DIR}/{ENTITY_TABLE}.parquet)
entity_embedding_df pd.read_parquet(f{INPUT_DIR}/{ENTITY_EMBEDDING_TABLE}.parquet)entities read_indexer_entities(entity_df, entity_embedding_df, COMMUNITY_LEVEL)# load description embeddings to an in-memory lancedb vectorstore
# to connect to a remote db, specify url and port values.
description_embedding_store LanceDBVectorStore(collection_nameentity_description_embeddings,
)
description_embedding_store.connect(db_uriLANCEDB_URI)
entity_description_embeddings store_entity_semantic_embeddings(entitiesentities, vectorstoredescription_embedding_store
)# 读取关系relationships
relationship_df pd.read_parquet(f{INPUT_DIR}/{RELATIONSHIP_TABLE}.parquet)
relationships read_indexer_relationships(relationship_df)# 读取协变量covariates
covariate_df pd.read_parquet(f{INPUT_DIR}/{COVARIATE_TABLE}.parquet)claims read_indexer_covariates(covariate_df)print(fClaim records: {len(claims)})
covariates {claims: claims}# 读取社区报告
report_df pd.read_parquet(f{INPUT_DIR}/{COMMUNITY_REPORT_TABLE}.parquet)
reports read_indexer_reports(report_df, entity_df, COMMUNITY_LEVEL)# 读取文本块
text_unit_df pd.read_parquet(f{INPUT_DIR}/{TEXT_UNIT_TABLE}.parquet)
text_units read_indexer_text_units(text_unit_df)# 配置模型参数
llm ChatOpenAI(api_keyollama,modelqwen2,api_basehttp://localhost:11434/v1/,api_typeOpenaiApiType.OpenAI, # OpenaiApiType.OpenAI or OpenaiApiType.AzureOpenAImax_retries20,
)token_encoder tiktoken.get_encoding(cl100k_base)text_embedder OpenAIEmbedding(api_keyollama,api_typeOpenaiApiType.OpenAI,api_basehttp://localhost:11434/v1/,modelqwen2,deployment_nameqwen2,max_retries20,
)# 创建局部搜索上下文构建器context-builder
context_builder LocalSearchMixedContext(community_reportsreports,text_unitstext_units,entitiesentities,relationshipsrelationships,covariatescovariates,entity_text_embeddingsdescription_embedding_store,embedding_vectorstore_keyEntityVectorStoreKey.ID, # if the vectorstore uses entity title as ids, set this to EntityVectorStoreKey.TITLEtext_embeddertext_embedder,token_encodertoken_encoder,
)# 创建局部搜索引擎
local_context_params {text_unit_prop: 0.5,community_prop: 0.1,conversation_history_max_turns: 5,conversation_history_user_turns_only: True,top_k_mapped_entities: 10,top_k_relationships: 10,include_entity_rank: True,include_relationship_weight: True,include_community_rank: False,return_candidate_context: False,embedding_vectorstore_key: EntityVectorStoreKey.ID, # set this to EntityVectorStoreKey.TITLE if the vectorstore uses entity title as idsmax_tokens: 12_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 5000)
}llm_params {max_tokens: 2_000, # change this based on the token limit you have on your model (if you are using a model with 8k limit, a good setting could be 10001500)temperature: 0.0,
}search_engine LocalSearch(llmllm,context_buildercontext_builder,token_encodertoken_encoder,llm_paramsllm_params,context_builder_paramslocal_context_params,response_typemultiple paragraphs, # free form text describing the response type and format, can be anything, e.g. prioritized list, single paragraph, multiple paragraphs, multiple-page report
)# 执行局部搜索
result await search_engine.asearch(叶文洁是谁)
print(result.response)# 查看local Search依赖的上下文:
print(result.context_data)运行效果
叶文洁是中国科幻作家刘慈欣的长篇科幻小说《三体》中的一个主要角色。在故事中她是一位资深的天文学家和物理学家在中国科学院从事研究工作。叶文洁在年轻时因政治原因遭受迫害后来成为“红卫兵”运动的积极参与者并因此被下放到农村劳动改造。在小说中她通过无线电波向宇宙发送了求救信号结果意外地接收到三体文明的信息从而引发了后续一系列惊心动魄的故事。叶文洁的性格复杂多面既有对科学和真理的执着追求也有对人性和社会的深刻洞察。她在故事中的经历反映了人类在面对未知、恐惧与希望之间的挣扎以及在极端环境下个人命运的脆弱性和坚韧性的交织。依赖的上下文:
{relationships: id source target description weight \0 4 小红卫兵 QUESTIONING 小红卫兵对叶哲泰的回答提出疑问试图理解是否有上帝的存在。)(entity 1.0 rank in_context 0 2 True ,claims: Empty DataFrameColumns: [in_context]Index: [],entities: id entity \0 52 会场 1 45 四位小将 2 26 琳 3 60 教工宿舍楼 4 51 帝 5 49 小红卫兵 6 53 宗教 7 41 实验结果 8 21 基础课 9 4 铁炉子 10 6 全国范围的武斗 11 15 批斗会 12 62 阮雯 13 61 阮老师 14 58 父亲 15 48 胡卫兵 16 19 批判(SPANEVENT/SPAN) 17 28 生态宇宙模型 18 56 组织 19 25 革命小将们 description number of relationships \0 会场是一个特定的地点可能是某个会议或集会的地方。 0 1 四位小将指的是来自附中的四位女性学生她们以一种坚定的方式进行“革命”通过实际行动表达... 0 2 琳是叶哲泰的妻子或女儿以其过人的天资和聪明才智著称在学术上有着重要的地位。 0 3 教工宿舍楼是叶文洁生活和工作的地点位于学校内。 0 4 帝是一个象征性的存在代表某种超自然或宇宙之外的力量。) (entity 0 5 小红卫兵对叶哲泰的回答感到困惑并试图理解是否有上帝的存在。 1 6 宗教在这里可能是指某种信仰体系被描述为被统治阶级用来控制人民的精神工具。 0 7 实验结果指的是与量子波函数坍缩相关的科学实验的结果。 0 8 基础课指的是教育体系中的一个课程或阶段涉及到物理学的基础理论教学。) (entity 0 9 铁炉子是一个充满烈性炸药的地方暗示了潜在的危险或冲突。) (entity 0 10 全国范围的武斗指的是在一个广泛区域内的武装冲突或斗争活动。) (entity 0 11 批斗会是一个几千人参加的事件在这个事件中人们聚集起来对一个反动学术权威进行批判。 0 12 阮雯是故事中的一个角色她拥有自己的家并且与叶文洁有关系。) (entity 0 13 阮老师是阮雯除父亲外最亲近的人在停课闹革命期间一直陪伴着她。) (entity 0 14 父亲是叶文洁的已故亲人她将烟斗放在了他的手中。 0 15 胡卫兵可能是一个与红卫兵相关的组织或群体但具体信息不明确。 0 16 批判指的是长时间的批评活动它在政治上产生了强烈的影响摧毁了参与者的意识和思想体系。参... 0 17 生态宇宙模型是一个被批判的概念因为它否认物质运动的本质被认为是反辩证法和反动唯心主义。 0 18 叶文洁是故事中的一个人物她与父亲叶哲泰有关联。) (entity 0 19 革命小将是帮助她醒悟并支持她的群体表明了他们对社会变革的支持和参与.) (entity 0 in_context 0 True 1 True 2 True 3 True 4 True 5 True 6 True 7 True 8 True 9 True 10 True 11 True 12 True 13 True 14 True 15 True 16 True 17 True 18 True 19 True ,sources: id text0 29 相信它不存在了。\n\n 这句大逆不道的话在整个会场引起了骚动在台上一名红卫兵的带领下...1 40 不讲。但来自附中的四位小将自有她们“无坚不摧”的革命方式刚才动手的那个女孩儿又狠抽了叶哲泰...2 21 态宇宙模型否定了物质的运动本性是反辩证法的它认为宇宙有限更是彻头彻尾的反动唯心主义...3 43 四肢仍保持着老校工抓着她时的姿态一动不动像石化了一般。过了好久她才将悬空的手臂放下来...4 28 帝的存在留下了位置。”绍琳对女孩儿点点头提示说。\n\n 小红卫兵那茫然的思路立刻找到了立...5 1 那一个她不由自主地问道 “连时间都是从那个奇点开始的那奇点以前有什么”\n\n ...6 39 神免于彻底垮掉。“叶哲泰这一点你是无法抵赖的你多次向学生散布反动的哥本哈根解释”\n\...7 17 二至六五届的基础课中你是不是擅自加入了大量的相对论内容”\n\n “相对论已经成为...8 3 铁炉子里面塞满了烈性炸药用电雷管串联起来他看不到它们但能感觉到它们磁石般的存在开关...9 5 全国范围的武斗也进入高潮。)——连同那些梭标和大刀等冷兵器构成了一部浓缩的近现代史……...10 9 场上一场几千人参加的批斗会已经进行了近两个小时。在这个派别林立的年代任何一处都有错综...11 34 们拿在手中和含在嘴里深思的那个男人的智慧但阮雯从未提起过他。这个雅致温暖的小世界成为文洁逃...12 45 来停课闹革命至今阮老师一直是她除父亲外最亲近的人。阮雯曾留学剑桥她的家曾对叶文洁充满了吸...13 41 动的一个”一名男红卫兵试图转移话题。\n\n “也许以后这个理论会被推翻但本世纪的两大...14 12 阶段旷日持久的批判将鲜明的政治图像如水银般注入了他们的意识将他们那由知识和理性构筑的思...15 42 这声音是精神已彻底崩溃的绍琳发出的听起来十分恐怖。人们开始离去最后发展成一场大溃逃...16 20 连其中的颤抖也放大了“你没有想到我会站出来揭发你批判你吧是的我以前受你欺骗你用...}使用ollama提供llm服务lm-studio提供Embedding服务运行GraphRAG的方法
注意如果使用lm-studio提供Embedding服务不需要修改这两个文件D:\ProgramData\miniconda3\envs\graphRAG\Lib\site-packages\graphrag\llm\openai\openai_embeddings_llm.py和D:\ProgramData\miniconda3\envs\graphRAG\Lib\site-packages\graphrag\query\llm\oai\embedding.py维持官方提供原始的样子
.env的修改同上 GRAPHRAG_API_KEYollama
GRAPHRAG_CLAIM_EXTRACTION_ENABLEDTruesetting.yaml的配置如下: encoding_model: cl100k_base
skip_workflows: []
llm:api_key: ollamatype: openai_chat # or azure_openai_chatmodel: qwen2model_supports_json: true # recommended if this is available for your model.# max_tokens: 4000# request_timeout: 180.0api_base: http://localhost:11434/v1/# api_version: 2024-02-15-preview# organization: organization_id# deployment_name: azure_model_deployment_name# tokens_per_minute: 150_000 # set a leaky bucket throttle# requests_per_minute: 10_000 # set a leaky bucket throttle# max_retries: 10# max_retry_wait: 10.0# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times# concurrent_requests: 25 # the number of parallel inflight requests that may be madeparallelization:stagger: 0.3# num_threads: 50 # the number of threads to use for parallel processingasync_mode: threaded # or asyncioembeddings:## parallelization: override the global parallelization settings for embeddingsasync_mode: threaded # or asynciollm:#api_key: ${GRAPHRAG_API_KEY}api_key: lm-studiotype: openai_embedding # or azure_openai_embeddingmodel: nomic-ai/nomic-embed-text-v1.5/nomic-embed-text-v1.5.Q8_0.ggufapi_base: http://localhost:1234/v1# api_version: 2024-02-15-preview# organization: organization_id# deployment_name: azure_model_deployment_name# tokens_per_minute: 150_000 # set a leaky bucket throttle# requests_per_minute: 10_000 # set a leaky bucket throttle# max_retries: 10# max_retry_wait: 10.0# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-timesconcurrent_requests: 1 # the number of parallel inflight requests that may be made# batch_size: 16 # the number of documents to send in a single request# batch_max_tokens: 8191 # the maximum number of tokens to send in a single request# target: required # or optional...
chunk_text()函数修改同上:
def chunk_text(text: str, max_tokens: int, token_encoder: tiktoken.Encoding | None None
):Chunk text by token length.if token_encoder is None:token_encoder tiktoken.get_encoding(cl100k_base)tokens token_encoder.encode(text) # type: ignoretokens token_encoder.decode(tokens) # 将tokens解码成字符串chunk_iterator batched(iter(tokens), max_tokens)yield from chunk_iterator
