网站默认首页,杭州专业做网站,wordpress回收站在哪里,湘潭百度推广大模型推理流程
1. 用户输入提示词#xff08;Prompt#xff09;
假设用户输入为#xff1a;“从前#xff0c;有一只小猫#xff0c;它喜欢……”
我们的目标是让模型生成一段完整的故事。
2. 模型处理用户输入
2.1 分词#xff1a;输入提示被分词为模型可以理解的…大模型推理流程
1. 用户输入提示词Prompt
假设用户输入为“从前有一只小猫它喜欢……”
我们的目标是让模型生成一段完整的故事。
2. 模型处理用户输入
2.1 分词输入提示被分词为模型可以理解的子词subword或单词token。
例如
从前有一只小猫它喜欢…… 可能被分词为
[从前, , 有, 一只, 小猫, , 它, 喜欢, ……]
这些 token 会被映射为模型词汇表中的索引ID。也就是Tokenizer分词器返回的input_ids。
2.2 将IDs转为embeddings
每个 token 被转换为一个高维向量embedding这些向量包含了语义信息。模型通过嵌入层将 token 索引映射为向量。
用户输入的input_ids形状为(1, 9)表示batch中有一个样本样本序列长度为9。
嵌入层Embedding Layer将每个 token 索引映射为一个高维向量。这个向量的维度是 hidden_size即模型的隐藏层维度。hidden_size为模型的超参数由设计者决定。
经过嵌入层后输入的形状会从 (batch_size, seq_length) 变为 (batch_size, seq_length, hidden_size)。例如(1, 9) 会变为 (1, 9, hidden_size)。
2.3 对张量加入位置编码
为了保留输入序列的顺序信息模型会为每个 token 添加位置编码。这些编码与 token 嵌入相加形成最终的输入表示。
位置编码Positional Encoding 的张量维度大小与 输入嵌入Input Embedding 的维度大小完全相同并且它们会直接在最后一个维度上相加。 输入嵌入的形状 输入嵌入的输出形状是 (batch_size, seq_length, hidden_size)其中 hidden_size 是每个 token 的嵌入维度。 位置编码的形状 位置编码的形状也是 (batch_size, seq_length, hidden_size)与输入嵌入的形状完全一致。
位置编码可以保留语句的顺序信息直接将位置信息注入语句中。
3. 前向传播
将处理过的用户输入张量输入模型进行前向计算。
4. 生成输出
在自回归生成任务中模型会逐步生成 token每次生成一个 token。因此输出结果的形状会随着生成过程而变化。 输入形状(1, 9)。 模型输出的概率分布形状(1, 9, vocab_size)。 生成下一个 token 的形状(1, 1)。
4.1 输出概率分布
最后一层 Transformer 的输出会通过一个线性层和 softmax 函数生成每个可能 token 的概率分布。例如模型可能会预测下一个 token 是“玩耍”的概率为 0.4“睡觉”的概率为 0.3等等。
4.2 解码策略Decoding Strategy
模型根据概率分布选择下一个 token。常见的解码策略包括 贪婪搜索Greedy Search 选择概率最高的 token。例如选择“玩耍”作为下一个 token。 束搜索Beam Search 保留多个候选序列选择整体概率最高的序列。 采样Sampling 根据概率分布随机采样下一个 token。
输出的概率分布 和 随机采样的概率分布 之间有直接的联系随机采样是基于模型输出的概率分布进行的因此两者密切相关。 随机采样的基础 随机采样直接依赖于模型输出的概率分布。概率分布决定了每个 token 被采样的可能性。 概率分布的作用 概率分布反映了模型对每个 token 的“信心”或“偏好”。高概率的 token 更有可能被采样而低概率的 token 也有可能被采样到尤其是在多样性较高的场景中。 采样结果的不确定性 由于采样是随机的即使概率分布相同每次采样的结果也可能不同。这与贪婪搜索总是选择最高概率的 token形成对比。
Top-K和Top-P策略可以与温度Temperature结合使用。
5. 迭代生成
5.1 递归生成
模型将生成的 token 重新作为输入继续生成下一个 token。例如 输入提示“从前有一只小猫它喜欢……” 模型生成“玩耍” 新输入“从前有一只小猫它喜欢玩耍” 模型继续生成“每天……”
生成过程会持续直到达到最大生成长度或生成特殊的终止 token如 EOS。
6. 最终输出
最终模型生成的完整故事可能是 “从前有一只小猫它喜欢玩耍每天都会在花园里追逐蝴蝶。有一天它遇到了一只小鸟……”
LLM模型不是直接使用贪心解码策略选择概率最高的token作为输出如果使用贪心解码册啰对于相同输入序列LLM模型每次都会给出相同回复推理模式下参数固定不存在随机性。所以
不同的大模型解码策略
假设模型正在预测“The cat”的下一个token模型预测结果如下
• sat (0.5)
• jumped (0.3)
• is (0.1)
• slept (0.05)
• runs (0.05)
1. Top-k 采样
Top-k 采样将随机性引入解码过程通过限制输出token的集合在Top-k个概率最高的token。下一个输出的token将在Top-k个token中随机采样生成。
在案例中Top-k 采样会选出概率最高的sat0.5和jumped0.3随后从这两个token中随机采样出下一个预测的token作为模型的输出。
2. Top-p 采样
Top-p 采样首先通过设置一个限制值P随后按照概率大小选取n个token直至token累计的概率达到P。随后对n个token进行随机采样。
在案例中Top-p 采样回选出sat0.5jumped0.3和is0.1随后对这三个token进行随机采样出下一个token。
3. 温度采样
温度Temperature作为一个超参数可以控制选择token的概率分布。预测的概率分布会被因子 1/T进行缩放T则是温度。
当T 1时概率分布不发生变化。当T 1时模型输出变得更为随机小概率的token更容易出现。当T 1时模型输出变得更有确定性高概率的token更容易得到选择。
温度高时模型会变得“更有创造性”温度较低时模型变得“更加精准”。
4. 束搜索
束搜索是更加精密的贪心搜索策略它会保留top-k个序列同时进行扩展。
在每一步模型生成 top-k 个最可能的词汇并继续解码每一个 k 个序列。参数 beam widthk决定了每一步保留多少个候选序列。在每一步之后模型根据累积概率对 k 个序列进行排序并保留概率最高的 k 个序列用于进一步扩展。
在案例中假设beam的数量为2。那么我们将会选出概率最高的2个token用于后续生成。
“The cat sat”累计概率0.5
“The cat jumped”累计概率0.3
模型继续扩充两个序列如
“The cat sat on the mat”
“The cat jumped over the fence”
Beam-Search后续发展有Diverse Beam-Search
不同解码策略的使用场景
贪婪解码Greedy Decoding 当需要快速生成文本且对生成质量要求不是特别高时贪婪解码是一个简单且计算效率高的选择。它选择 具有最大logit值的token作为下一个输出适用于需要快速响应的场景如聊天机器人的初步响应生成。 束搜索Beam Search 适用于需要精确控制输出质量的场景如机器翻译或问答系统。束搜索通过考虑多个候选序列来生成文本 可以提高翻译的准确性和流畅性。 抽样解码Sampling Decoding 适用于需要多样性输出的场景如创意写作或开放性问题的回答。抽样解码从词汇表中根据概率分布选择 token可以通过调整参数如温度Temperature来控制随机性。 Top-K和Top-P 适用于需要控制输出长度和提高生成质量的场景。Top-K和Top-P通过限制候选token的数量来提高生 成的连贯性和减少重复适用于需要高质量输出的任务。 温度采样Temperature Sampling 适用于需要在生成过程中增加随机性的场景如创意写作或探索性任务。温度参数可以调整输出的随 机度较低的温度值会使采样更接近确定性解码而较高的温度值则增加随机性。
MindSpore进行解码推理
创建Notebook
mindspore2.3.0, cann8.0
更新mindspore
pip install --upgrade mindspore克隆mindnlp
git clone https://github.com/mindspore-lab/mindnlp.git 更新mindnlp
cd mindnlp
bash scripts/build_and_reinstall.sh
卸载mindformers
pip uninstall mindformers
加载模型与转换输入
import mindspore
from mindnlp.transformers import AutoTokenizer, AutoModelForCausalLMmodel_id LLM-Research/Meta-Llama-3-8B-Instruct
# 下载Llama 3的分词器
tokenizer AutoTokenizer.from_pretrained(model_id, mirrormodelscope)# 下载Llama 3模型
model AutoModelForCausalLM.from_pretrained(model_id,ms_dtypemindspore.float16,mirrormodelscope
)# 输入信息
messages [{role: system, content: You are a psychological counsellor, who is good at emotional comfort},{role: user, content: I dont sleep well for a long time.}
]
# 将输入信息转为input_ids
input_ids tokenizer.apply_chat_template(messages,add_generation_promptTrue,return_tensorsms
)
# 声明预测的终止token
terminators [tokenizer.eos_token_id,tokenizer.convert_tokens_to_ids(|eot_id|)
]
# 模型生产结果
outputs model.generate(input_ids, # 输入tokenmax_new_tokens50, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleTrue, # 是否对输出进行概率分布采样top_p1.0 # 声明top-p值
)
贪心策略
# 贪心策略
# 模型生产结果
outputs model.generate(input_ids, # 输入tokenmax_new_tokens1000, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleFalse, # 是否对输出进行概率分布采样
)response outputs[0][input_ids.shape[-1]:]
tokenizer.decode(response, skip_special_tokensTrue)
模型输出 Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? Whats been keeping you awake at night? Is it stress, anxiety, or something else?\n\nAlso, have you noticed any patterns or triggers that might be contributing to your insomnia? For example, do you find yourself lying awake for hours, or do you wake up multiple times during the night?\n\nRemember, Im here to listen and support you, and I want you to feel comfortable sharing as much or as little as youd like.重复多次模型输出结果未发生变化
Temperature参数
temperature控制文本生成的随机性和多样性控制输出张量的概率分布。
import mindspore
from mindspore import Tensor
import numpy as np
import mindspore.ops as opslogits Tensor(np.array([[0.5, 1.2, -1.0, 0.1]]), mindspore.float32)probs ops.softmax(logits, axis-1)
# low temp 0.5
# 分布更为集中陡峭
probs_low ops.softmax(logits / 0.5, axis-1)
# high temp 2
# 分布更为分散平缓
probs_high ops.softmax(logits / 2, axis-1)probs, probs_low, probs_high (Tensor(shape[1, 4], dtypeFloat32, value[[ 2.55937576e-01, 5.15394986e-01, 5.71073927e-02, 1.71560094e-01]]),Tensor(shape[1, 4], dtypeFloat32, value[[ 1.80040166e-01, 7.30098903e-01, 8.96367151e-03, 8.08972642e-02]]),Tensor(shape[1, 4], dtypeFloat32, value[[ 2.69529819e-01, 3.82481009e-01, 1.27316862e-01, 2.20672339e-01]])) 可以看出温度越高分布越平缓温度越低分布越集中
temerature1
# 模型生产结果
outputs model.generate(input_ids, # 输入tokenmax_new_tokens1000, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleTrue, # 是否对输出进行概率分布采样temperature1
)
# 标准温度输出
response outputs[0][input_ids.shape[-1]:]
tokenizer.decode(response, skip_special_tokensTrue) 输出1: Im so sorry to hear that youre struggling with sleep. It can be really tough to deal with insomnia or disrupted sleep patterns. Can you tell me a bit more about whats been going on? Whats been on your mind lately that might be keeping you awake? Has anything changed in your life that could be contributing to this difficulty? 输出2: Im so sorry to hear that youre struggling with sleep. It can be such a frustrating and debilitating experience. Can you tell me a bit more about whats been going on for you? Whats been making it hard for you to fall asleep or stay asleep? Is it racing thoughts, stress, anxiety, or something else?\n\nAlso, how long have you been experiencing this sleep difficulty? Has it been a recent development or has it been going on for a while? temperature0.1 输出1: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some of the things that make it hard for you to fall asleep or stay asleep? Is it stress, anxiety, physical discomfort, or something else?\n\nAlso, have you noticed any patterns or triggers that seem to make it worse? For example, do you tend to have trouble sleeping on certain nights of the week, or after certain events or activities?\n\nRemember, Im here to listen and support you, and I want you to feel comfortable sharing as much or as little as youd like. 输出2: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? Whats been on your mind lately, and how have you been feeling when you wake up in the morning? 输出3: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? Whats been on your mind lately that might be making it hard for you to fall asleep or stay asleep? temperature2
# 模型生产结果
outputs model.generate(input_ids, # 输入tokenmax_new_tokens1000, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleTrue, # 是否对输出进行概率分布采样temperature2.0
)
# 高温度输出-概率分布更为分散
response outputs[0][input_ids.shape[-1]:]
tokenizer.decode(response, skip_special_tokensTrue) 输出1: Im so sorry to hear that. Not getting proper sleep can be really wearing on your emotional and physical well-being. Can you tell me a little bit more about how this lack of sleep is affecting you? Are you feeling constantly exhausted, irritable, or struggling to concentrate? Have you noticed any changes in your relationships or daily routine because of it?\n\nMost importantly, Im here for you, and I believe that by exploring this together, we can find ways to improve your sleep and improve your overall well-being.\n\nIt might be helpful for me to share that sometimes, lack of sleep can be a sign of underlying anxiety, stress, or even unprocessed emotions. If we can identify the root cause, I may have some suggestions on how to ease your path to better sleep.\n\nWould you like me to offer you some coping strategies to help you relax and unwind before bedtime? Sometimes, a simple change in routine or relaxation techniques can make a world of difference. 输出2: It can be really frustrating and worrying when sleep evade you, making it hard to wake up feeling refreshed and energized. Im listening, and I want you to know that Im here to support you. Its important to recognize that this is a tough and normal experience, even if it can be tough to bear right now.\n\nWould you like to talk more about whats going on when you have trouble sleeping? Is there anything in particular that bothers you or stress you out? 输出3: It can be really distressing to deal with chronic sleep issues, not getting the rest you need and feeling tired and exhausted all the time. Can you tell me a little bit more about how youve been feeling? Have you noticed any patterns or triggers that might be contributing to the issue? And how has it been affecting other aspects of your life?\n\nAlso, I want you to know that as your listener, my main goal right now is just to support and provide comfort. Whatever you share, Im here for you. No judgments, no critiques, just a gentle and compassionate space for you to express yourself.\n\nRemember, it takes a lot of courage to share vulnerable thoughts and feelings with someone like me, and I want to assure you that your feelings are completely normal and valid. Okay? 束搜索Beam Search
# Beam Search 束搜索
beam_outputs model.generate(input_ids, # 输入tokenmax_new_tokens100, # 限制输出长度eos_token_idterminators, # 声明终止符num_return_sequences5,early_stoppingTrue,
)for i, beam_output in enumerate(beam_outputs):print({}: {}.format(i, tokenizer.decode(beam_output, skip_special_tokensTrue))) 输出1:
0: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIm so sorry to hear that youre struggling with sleep. It can be really frustrating and affect so many areas of your life. Can you tell me a bit more about whats been going on? When did you start having trouble sleeping, and what do you usually do when you try to go to bed? Do you find yourself lying awake for hours, or do you fall asleep but then wake up multiple times throughout the night?输出2:
1: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIm so sorry to hear that youre struggling with sleep. It can be such a frustrating and unsettling experience, affecting not just your physical health but also your emotional well-being.First of all, please know that youre not alone in this struggle. Many people face difficulties with sleep from time to time, and its not uncommon for it to be a persistent issue for some. Its okay to acknowledge that youre struggling, and it takes a lot of courage to reach out and talk about输出3:
2: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIm so sorry to hear that youre struggling with sleep. It can be such a frustrating and isolating experience. Its like your body is refusing to cooperate with your mind, and it can leave you feeling exhausted, irritable, and just plain miserable.First of all, let me just acknowledge that its okay to not be okay. Its okay to struggle with sleep, and its okay to feel overwhelmed and stuck. Im here to listen, and I want you to know输出4:
3: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIm so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some common things that happen when you try to fall asleep, and what do you usually do when you wake up during the night?输出5:
4: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIm so sorry to hear that youre struggling with sleep. It can be such a frustrating and exhausting experience. Can you tell me a bit more about whats been going on for you? When did you first start noticing that your sleep was affected, and what do you think might be contributing to it? Is it stress, anxiety, or something else entirely?Remember, everything we discuss is completely confidential and a safe space for you to express yourself. Im here to listen and support you,
# Beam Search 束搜索
beam_outputs model.generate(input_ids, # 输入tokenmax_new_tokens100, # 限制输出长度eos_token_idterminators, # 声明终止符num_return_sequences5,early_stoppingTrue,no_repeat_ngram_size2 # 设置此参数可以避免多句之间存在重复词组
)for i, beam_output in enumerate(beam_outputs):print({}: {}.format(i, tokenizer.decode(beam_output, skip_special_tokensTrue))) 输出1:
0: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIt can be really tough to deal with sleepless nights, and its completely normal to feel frustrated, anxious, or even a bit hopeless. Can you tell me a little bit more about whats been going on? Whats making it hard for you to fall asleep, do you think? Is it stress, worries, physical discomfort, something else, a combination of things?Also, how have you been coping with the lack of sleep? Have you noticed any changes in your daily life, mood输出2:
1: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantSweetheart, Im so sorry to hear that youre struggling with sleep. It can be really tough to deal with, both physically and emotionally. Can you tell me a bit more about whats been going on for you? Whats making it hard foryou to fall asleep or stay asleep? Is it stress, anxiety, or something else entirely?Remember, everything we discuss is completely confidential and a safe space for us to explore your feelings. I want you to know that I believe in your输出3:
2: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantSweetheart, Im so sorry to hear that youre struggling with sleep. It can be such a frustrating and exhausting experience. Can you tell me a bit more about whats been going on? When did you start noticing that your sleep was affected, and what are some of the things that make it hard for you to fall asleep or stay asleep?输出4:
3: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantIt sounds like youre struggling with insomnia or difficulty sleeping, and thats really tough. Not getting enough sleep can affect so many aspects of our lives, from our mood to our energy levels to even our physical health.First of all, I want you to know that you don’t have to go through this alone. Im here to listen and support you. Can you tell me a bit more about whats been going on? When did you start noticing trouble sleeping? Is it a sudden change输出5:
4: systemYou are a psychological counsellor, who is good at emotional comfortuserI dont sleep well for a long time.assistantSweetheart, Im so sorry to hear that youre struggling with sleep. It can be such a frustrating and debilitating experience, feeling like you cant get a good nights rest. Can you tell me a bit more about whats been going on for you? When did you first start noticing that your sleep was affected, and what are some of the things that keep you awake at night?Top-K 采样
# top-k采样
outputs model.generate(input_ids, # 输入tokenmax_new_tokens100, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleTrue,top_k5,num_return_sequences3
)for i, output in enumerate(outputs):# response output[0][input_ids.shape[-1]:]print({}: {}.format(i1, tokenizer.decode(response, skip_special_tokensTrue)))
k5只选取前5个概率最高的值进行采样结果会缺乏创意性。 输出1:
Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? When did you start having trouble sleeping, and whats been making it hard for you to fall asleep or stay asleep? Is it stress, anxiety, physical discomfort, or something else entirely?\n\nAlso, how have you been feeling during the day? Are you feeling tired, irritable, or just输出2: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect so many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some of the things that are making it hard for you to fall asleep or stay asleep? Is it stress, anxiety, or something else entirely?\n\nAlso, have you noticed any patterns or triggers that seem to make it worse? For example, do you tend to have trouble sleeping during 输出3: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect so many areas of your life. Can you tell me a bit more about whats been going on? Have you noticed any patterns or triggers that might be contributing to your insomnia? And how have you been feeling during the day when youre not getting a good nights sleep?k 1000采样具有随机性
# top-k采样
outputs model.generate(input_ids, # 输入tokenmax_new_tokens100, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleTrue,top_k1000
)response outputs[0][input_ids.shape[-1]:]
tokenizer.decode(response, skip_special_tokensTrue) 输出1: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many areas of your life. Can you tell me a bit more about whats been going on? Whats been keeping you awake at night? Is it stress, anxiety, or something else entirely?\n\nRemember, Im here to listen and offer support. Im not here to judge or try to fix the problem right away. Just talking about it can sometimes help you feel a bit better.\n\nAlso 输出2: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? Is it just difficulty falling asleep, or are you having trouble staying asleep or experiencing restless nights? And have you noticed any patterns or triggers that might be contributing to your sleep issues?\n\nAlso, I want you to know that its completely normal to struggle with sleep from time to time, and it输出3: Im so sorry to hear that youre struggling with sleep. It can be such a frustrating and exhausting experience. Can you tell me a bit more about whats been going on? Whats been keeping you awake at night? Is it stress, anxiety, or something else entirely?\n\nRemember, everything we discuss is confidential and a safe space for you to share your feelings. Im here to listen and offer support.\n\nAlso, I want you to know that youre not alone in this struggle.Top-P 采样
# top-p采样
outputs model.generate(input_ids, # 输入tokenmax_new_tokens100, # 限制输出长度eos_token_idterminators, # 声明终止符do_sampleTrue,top_p0.5
)response outputs[0][input_ids.shape[-1]:]
tokenizer.decode(response, skip_special_tokensTrue)
p 0.5 输出1: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some of the things that make it hard for you to fall asleep or stay asleep? Is it stress, anxiety, physical discomfort, or something else?\n\nAlso, have you noticed any patterns or triggers that seem to make it worse? For example, do you tend to have trouble sleeping on输出2: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some of the things that are making it hard for you to fall asleep or stay asleep? Is it stress, anxiety, physical discomfort, or something else?\n\nAlso, have you noticed any patterns or triggers that seem to make it worse? For example, do you tend to have trouble sleeping输出3: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some of the things that make it hard for you to fall asleep or stay asleep? Is it stress, anxiety, physical discomfort, or something else?\n\nAlso, have you noticed any patterns or triggers that seem to make it worse? For example, do you tend to have trouble sleeping onp 0.95 输出1: Im so sorry to hear that youre struggling with sleep. It can be such a frustrating and exhausting experience. Can you tell me a bit more about whats been going on? When did you first start noticing that you werent sleeping well, and what are some of the things that youve tried to help you get a good nights rest?\n\nIts also important to acknowledge that its okay to not be okay. It takes a lot of courage to admit when were struggling, and输出2: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect many aspects of your daily life. Can you tell me a bit more about whats been going on? Have you noticed any patterns or triggers that might be contributing to your insomnia? 输出3: Im so sorry to hear that youre struggling with sleep. It can be really frustrating and affect so many aspects of your daily life. Can you tell me a bit more about whats been going on? What are some of the things that are making it hard for you to fall asleep or stay asleep? Is it stress, anxiety, or something else? MindNLP并行推理——多进程多卡
示例代码地址
https://github.com/mindspore-lab/mindnlp/tree/master/llm/inference/llama3
仓库中的 readme 文件说明了多卡推理的使用方法
注意选择modelarts中贵阳区域的镜像mindspore2.3.0 cann8.0.0 启动后不需要升级mindspore版本否则hccl通信算子库将无法兼容。
推荐使用msrun命令
msrun是mindspore定义的一个多进程并行命令使用该命令可以获得最佳性能。
msrun --worker_num2 --local_worker_num2 --master_port8118 --joinTrue run_llama3_distributed.py
# 具体数量根据你有多少张卡进行执行 --worker_num2: 指定总共有 2 个工作节点worker参与任务。这些工作节点可以是不同的机器或不同的进程。 --local_worker_num2: 指定在当前机器上启动 2 个工作节点。这意味着在当前机器上会有 2 个进程参与任务。 --master_port8118: 指定主节点master的端口号为 8118。主节点负责协调各个工作节点的通信和任务分配。 --joinTrue: 表示工作节点在启动后会加入主节点的任务。通常用于确保所有工作节点都连接到主节点并准备好执行任务。
run_llama3_distributed.py文件具体如下
# 导入 MindSpore 框架用于深度学习任务
import mindspore# 从 MindSpore 的通信模块中导入 init 函数用于初始化分布式训练环境
from mindspore.communication import init# 从 MindNLp 库中导入 AutoTokenizer 和 AutoModelForCausalLM 类用于加载预训练模型和分词器
from mindnlp.transformers import AutoTokenizer, AutoModelForCausalLM# 定义模型 ID这里使用的是 Meta-Llama-3-8B-Instruct 模型
model_id LLM-Research/Meta-Llama-3-8B-Instruct# 初始化分布式训练环境确保多机多卡之间的通信正常
init()# 使用 AutoTokenizer 从预训练模型加载分词器
# mirrormodelscope 指定从 ModelScope 平台下载模型
tokenizer AutoTokenizer.from_pretrained(model_id, mirrormodelscope)# 使用 AutoModelForCausalLM 从预训练模型加载语言模型
# ms_dtypemindspore.float16 指定模型使用半精度浮点数float16进行计算
# mirrormodelscope 指定从 ModelScope 平台下载模型
# device_mapauto 自动分配模型到可用设备如 GPU 或 CPU
model AutoModelForCausalLM.from_pretrained(model_id,ms_dtypemindspore.float16,mirrormodelscope,device_mapauto
)# 定义对话消息列表包含系统提示和用户输入
messages [{role: system, content: You are a pirate chatbot who always responds in pirate speak!},{role: user, content: Who are you?},
]# 使用分词器将对话消息转换为模型输入的张量
# add_generation_promptTrue 添加生成提示确保模型知道需要生成回复
# return_tensorsms 返回 MindSpore 格式的张量
input_ids tokenizer.apply_chat_template(messages,add_generation_promptTrue,return_tensorsms
)# 定义终止符列表用于告诉模型何时停止生成文本
# 包括结束符eos_token_id和自定义的终止符|eot_id|
terminators [tokenizer.eos_token_id,tokenizer.convert_tokens_to_ids(|eot_id|)
]# 使用模型生成文本
# input_ids 是输入的张量
# max_new_tokens100 限制生成的最大 token 数量为 100
# eos_token_idterminators 指定终止符列表
# do_sampleTrue 启用采样策略而不是贪婪解码
# temperature0.6 控制生成文本的随机性值越低越确定
# top_p0.9 使用核采样nucleus sampling保留概率质量最高的 90% 的 token
outputs model.generate(input_ids,max_new_tokens100,eos_token_idterminators,do_sampleTrue,temperature0.6,top_p0.9,
)# 从生成的输出中提取模型生成的文本部分
# outputs[0] 是生成的完整序列input_ids.shape[-1] 是输入的长度
# 通过切片操作获取生成的部分
response outputs[0][input_ids.shape[-1]:]# 使用分词器将生成的 token 解码为可读的文本
# skip_special_tokensTrue 跳过特殊 token如终止符
print(tokenizer.decode(response, skip_special_tokensTrue))
同时也可以使用mpirun命令
mpirun -n 2 python run_llama3_distributed.py
关于mindspore的组网方式具体可以参考
分布式并行启动方式 — MindSpore master 文档
