专业电商网站,设计网站大全下载,便宜网站制作,5000元做网站值么文章目录 前言一、GLM-4v-9b模型model加载源码解读1、GLM-4v-9b模型model加载主函数源码2、GLM-4v-9b模型model加载源码源码解读3、GLM-4v-9b自定义模型类源码解读 二、基于GLM-4v-9b模型获取模型输入参数等内容源码解读(from_pretrained--huggingface)1、from_pretrained函… 文章目录 前言一、GLM-4v-9b模型model加载源码解读1、GLM-4v-9b模型model加载主函数源码2、GLM-4v-9b模型model加载源码源码解读3、GLM-4v-9b自定义模型类源码解读 二、基于GLM-4v-9b模型获取模型输入参数等内容源码解读(from_pretrained--huggingface)1、from_pretrained函数参数解读(huggingface)2、config文件路径获取(huggingface)3、adapter_config文件路径获取与加载(huggingface)4、config文件方法获取1、config源码2、from_pretrained函数kwargs参数传递3、config内容获取config加工源码调用AutoConfig.from_pretrained函数源码内容AutoConfig.from_pretrained的PretrainedConfig.get_config_dict函数源码解读config的类class加载源码解读 4、小节总结 前言
清华智普的GLM-4v-9b模型作为优化的多模态大模型特别适用于国内应用场景解决了国外模型本地化不足的问题。本专栏提供环境安装、数据处理、视觉与语言模型源码理解并基于Hugging Face重构GLM模型搭建教程帮助理解、修改和应用GLM墨西哥指导搭建多模态大模型帮助读者自由搭建与修改大模型。本节给出GLM-4-9B模型加载相关参数获取方法源码解读内容。
第一节GLM-4v-9B大模型安装、推理与训练详细教程 第二节GLM-4v-9B数据加载源码解读 第三节GLM-4v-9B数据加载之huggingface数据加载方法教程(通用大模型数据加载实列) 第四节GLM-4v-9b模型的tokenizer源码解读 第五节GLM-4v-9b模型model加载源码解读(模型相关参数方法解读) 第六节GLM-4v-9b模型加载源码解读(模型加载方法解读) 第七节GLM-4v-9b模型的视觉模型源码解读 第八节GLM-4v-9b模型的大语言模型源码解读(ChatGLMForConditionalGeneration) 第九节通过Debug解析ChatGLMForConditionalGeneration的数据流理解GLM-4v-9b模型架构 第十节通过Debug解析ChatGLMModel的数据流理解视觉与语言模型结合架构 第十一节利用huggingface重构GLM-4v-9B模型数据处理代码Demo 第十二节利用huggingface重构GLM-4v-9B训练模型代码Demo 第十一、十二节是在理解GLM-4v-9B模型后使用huggignface重新构建/搭建GLM-4v-9B模型使读者能自由构建多模态大模型
本节给出GLM-4v-9b模型model加载而加载使用huggingface方法是十分简单然而huggingface源码确为麻烦。为此本节以glm的model加载示例重点解读huggingface加载model源码。该小节主要解释模型加载前准备工作特别需要知道使用下面代码可以完成自定义模型加载 auto_map: {AutoConfig: configuration_chatglm.ChatGLMConfig,AutoModel: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForCausalLM: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForSeq2SeqLM: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForSequenceClassification: modeling_chatglm.ChatGLMForSequenceClassification},一、GLM-4v-9b模型model加载源码解读
我们解读下model加载流程吧实际这个部分与huggingface相关我们先给出与huggingface加载相关源码
1、GLM-4v-9b模型model加载主函数源码
来源finetune_demo/finetune_vision.py--load_tokenizer_and_model函数 我给出GLM-4v-9b模型加载函数其主要是调用huggingface内容我给出主函数加载源码如下
def load_tokenizer_and_model(model_dir: str,peft_config: Optional[PeftConfig] None,
):tokenizer AutoTokenizer.from_pretrained(model_dir, trust_remote_codeTrue)if peft_config is not None:model AutoModelForCausalLM.from_pretrained(model_dir,trust_remote_codeTrue,empty_initFalse,use_cacheFalse,torch_dtypetorch.bfloat16, # Must use BFloat 16# cache_dir /extend_disk/disk3/tj/GLM-4V-9B/GLM-4-main/THUDM/cache_dir)model get_peft_model(model, peft_config)model.print_trainable_parameters()else:model AutoModelForCausalLM.from_pretrained(model_dir,trust_remote_codeTrue,empty_initFalse,use_cacheFalse,torch_dtypetorch.bfloat16)return tokenizer, model2、GLM-4v-9b模型model加载源码源码解读
来源finetune_demo/finetune_vision.py--load_tokenizer_and_model函数 实际模型使用huggingface的AutoModelForCausalLM.from_pretrained来构建而get_peft_model是高效微调使用函数也就是模型加载也是比较简单其主要还是huggingface内容源码解读我将后面重点解读而GLM-4v-9b源码如下:
if peft_config is not None:model AutoModelForCausalLM.from_pretrained(model_dir,trust_remote_codeTrue,empty_initFalse,use_cacheFalse,torch_dtypetorch.bfloat16 # Must use BFloat 16)model get_peft_model(model, peft_config)model.print_trainable_parameters()
else:model AutoModelForCausalLM.from_pretrained(model_dir,trust_remote_codeTrue,empty_initFalse,use_cacheFalse,torch_dtypetorch.bfloat16)3、GLM-4v-9b自定义模型类源码解读
在权重文件中有个config.json文件这也是huggingface最重要配置文件。该文件有个参数auto_map字典里面包含了很多方法和之前tokenizer加载自定义方法有点类似。auto_map有AutoModel等内容用来加载模型列如AutoModel: modeling_chatglm.ChatGLMForConditionalGeneration就是在modeling_chatglm.py文件中加载 ChatGLMForConditionalGeneration类。我给出json这个文件内容如下。 auto_map: {AutoConfig: configuration_chatglm.ChatGLMConfig,AutoModel: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForCausalLM: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForSeq2SeqLM: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForSequenceClassification: modeling_chatglm.ChatGLMForSequenceClassification},我也给出GLM-4v-9b自定义ChatGLMForConditionalGeneration类部分内容如下
class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):def __init__(self, config: ChatGLMConfig, empty_initTrue, deviceNone):super().__init__(config)self.max_sequence_length config.max_lengthself.transformer ChatGLMModel(config, empty_initempty_init, devicedevice)self.config configdef _update_model_kwargs_for_generation(self,outputs: ModelOutput,model_kwargs: Dict[str, Any],is_encoder_decoder: bool False,) - Dict[str, Any]:# update past_key_valuescache_name, cache self._extract_past_from_model_output(outputs)model_kwargs[cache_name] cache# update attention maskif attention_mask in model_kwargs:attention_mask model_kwargs[attention_mask]model_kwargs[attention_mask] torch.cat([attention_mask, attention_mask.new_ones((attention_mask.shape[0], 1))], dim-1)# update position idsif position_ids in model_kwargs:position_ids model_kwargs[position_ids]new_position_id position_ids[..., -1:].clone()new_position_id 1model_kwargs[position_ids] torch.cat([position_ids, new_position_id], dim-1)model_kwargs[is_first_forward] Falsereturn model_kwargs二、基于GLM-4v-9b模型获取模型输入参数等内容源码解读(from_pretrained–huggingface)
来源transforms/models/auto/auto_factory.py--class _BaseAutoModelClass--from_pretrained函数
1、from_pretrained函数参数解读(huggingface)
这里我需要说明from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)参数特别是kwargs可以直接提供对应参数如config文件而from_pretrained内部如下图
2、config文件路径获取(huggingface)
通过以下代码就可以获取config文件路径给了resolved_config_file变量
if commit_hash is None:if not isinstance(config, PretrainedConfig):# We make a call to the config file first (which may be absent) to get the commit hash as soon as possibleresolved_config_file cached_file(pretrained_model_name_or_path,CONFIG_NAME,_raise_exceptions_for_gated_repoFalse,_raise_exceptions_for_missing_entriesFalse,_raise_exceptions_for_connection_errorsFalse,**hub_kwargs,)commit_hash extract_commit_hash(resolved_config_file, commit_hash)else:commit_hash getattr(config, _commit_hash, None)
如图显示如下
3、adapter_config文件路径获取与加载(huggingface)
这个代码是类似lora方法获得内容加载首先是加载adapter_config.josn文件该文件也是在pretrained_model_name_or_path路径中存在使用find_adapter_config_file函数来获取adapter_config.json路径其源码如下
if is_peft_available():if adapter_kwargs is None:adapter_kwargs {}if token is not None:adapter_kwargs[token] tokenmaybe_adapter_path find_adapter_config_file(pretrained_model_name_or_path, _commit_hashcommit_hash, **adapter_kwargs)if maybe_adapter_path is not None:with open(maybe_adapter_path, r, encodingutf-8) as f:adapter_config json.load(f)adapter_kwargs[_adapter_model_path] pretrained_model_name_or_pathpretrained_model_name_or_path adapter_config[base_model_name_or_path]如果存在adapter_config.json路径就执行with open(maybe_adapter_path, r, encodingutf-8) as f 获得其参数。
4、config文件方法获取
1、config源码
我们给出整体源码代码如下
if not isinstance(config, PretrainedConfig):kwargs_orig copy.deepcopy(kwargs)# ensure not to pollute the config object with torch_dtypeauto - since its# meaningless in the context of the config object - torch.dtype values are acceptableif kwargs.get(torch_dtype, None) auto:_ kwargs.pop(torch_dtype)# to not overwrite the quantization_config if config has a quantization_configif kwargs.get(quantization_config, None) is not None:_ kwargs.pop(quantization_config)config, kwargs AutoConfig.from_pretrained(pretrained_model_name_or_path,return_unused_kwargsTrue,trust_remote_codetrust_remote_code,code_revisioncode_revision,_commit_hashcommit_hash,**hub_kwargs,**kwargs,)# if torch_dtypeauto was passed here, ensure to pass it onif kwargs_orig.get(torch_dtype, None) auto:kwargs[torch_dtype] autoif kwargs_orig.get(quantization_config, None) is not None:kwargs[quantization_config] kwargs_orig[quantization_config]kwargs存在quantization_config与torch_dtype会直接剔除共下面函数调用。
2、from_pretrained函数kwargs参数传递
仅接着from_pretrained函数的参数kwargs将传递如下代码:
kwargs_orig copy.deepcopy(kwargs)3、config内容获取
config加工源码调用
获得config函数使用以下源码如下
config, kwargs AutoConfig.from_pretrained(pretrained_model_name_or_path, # 模型路径return_unused_kwargsTrue,trust_remote_codetrust_remote_code,code_revisioncode_revision,_commit_hashcommit_hash,**hub_kwargs,**kwargs,)
AutoConfig.from_pretrained函数源码内容
来源transforms/models/auto/configuration_auto.py--class AutoConfig--from_pretrained函数 我们继续解读AutoConfig.from_pretrained函数所有源码如下
classmethod
replace_list_option_in_docstrings()
def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):r...use_auth_token kwargs.pop(use_auth_token, None)if use_auth_token is not None:warnings.warn(The use_auth_token argument is deprecated and will be removed in v5 of Transformers. Please use token instead.,FutureWarning,)if kwargs.get(token, None) is not None:raise ValueError(token and use_auth_token are both specified. Please set only the argument token.)kwargs[token] use_auth_tokenkwargs[_from_auto] Truekwargs[name_or_path] pretrained_model_name_or_pathtrust_remote_code kwargs.pop(trust_remote_code, None)code_revision kwargs.pop(code_revision, None)config_dict, unused_kwargs PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)has_remote_code auto_map in config_dict and AutoConfig in config_dict[auto_map]has_local_code model_type in config_dict and config_dict[model_type] in CONFIG_MAPPINGtrust_remote_code resolve_trust_remote_code(trust_remote_code, pretrained_model_name_or_path, has_local_code, has_remote_code)if has_remote_code and trust_remote_code:class_ref config_dict[auto_map][AutoConfig]config_class get_class_from_dynamic_module(class_ref, pretrained_model_name_or_path, code_revisioncode_revision, **kwargs)if os.path.isdir(pretrained_model_name_or_path):config_class.register_for_auto_class()return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)elif model_type in config_dict:try:config_class CONFIG_MAPPING[config_dict[model_type]]except KeyError:raise ValueError(fThe checkpoint you are trying to load has model type {config_dict[model_type]} but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.)return config_class.from_dict(config_dict, **unused_kwargs)else:# Fallback: use pattern matching on the string.# We go from longer names to shorter names to catch roberta before bert (for instance)for pattern in sorted(CONFIG_MAPPING.keys(), keylen, reverseTrue):if pattern in str(pretrained_model_name_or_path):return CONFIG_MAPPING[pattern].from_dict(config_dict, **unused_kwargs)raise ValueError(fUnrecognized model in {pretrained_model_name_or_path}. fShould have a model_type key in its {CONFIG_NAME}, or contain one of the following strings fin its name: {, .join(CONFIG_MAPPING.keys())})AutoConfig.from_pretrained的PretrainedConfig.get_config_dict函数源码解读
简单说就是读取路径config文件内容重要代码如下
config_dict, unused_kwargs PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)传递路径pretrained_model_name_or_path变量中通过PretrainedConfig.get_config_dict函数可以直接获得config.json文件内容与其它unused_kwargs 内容而unused_kwargs就是kwargs内容。如下图所示 而unused_kwargs来源kwargs但有些参数被剔除了pop如下图所示 config的类class加载源码解读
这里就很关键了我们需要加载config对应class方法了当然你可以自己继承huggingface构建一个自己想处理的config方法class也可以调用已有的。我们来看下源代码如下
if has_remote_code and trust_remote_code:class_ref config_dict[auto_map][AutoConfig]config_class get_class_from_dynamic_module(class_ref, pretrained_model_name_or_path, code_revisioncode_revision, **kwargs)if os.path.isdir(pretrained_model_name_or_path):config_class.register_for_auto_class()return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)
elif model_type in config_dict:try:config_class CONFIG_MAPPING[config_dict[model_type]]except KeyError:raise ValueError(fThe checkpoint you are trying to load has model type {config_dict[model_type]} but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.)return config_class.from_dict(config_dict, **unused_kwargs)
else:# Fallback: use pattern matching on the string.# We go from longer names to shorter names to catch roberta before bert (for instance)for pattern in sorted(CONFIG_MAPPING.keys(), keylen, reverseTrue):if pattern in str(pretrained_model_name_or_path):return CONFIG_MAPPING[pattern].from_dict(config_dict, **unused_kwargs)①、如果config存在auto_map字典的AutoConfig内容 config.json的auto_map如下所示内容 auto_map: {AutoConfig: configuration_chatglm.ChatGLMConfig,AutoModel: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForCausalLM: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForSeq2SeqLM: modeling_chatglm.ChatGLMForConditionalGeneration,AutoModelForSequenceClassification: modeling_chatglm.ChatGLMForSequenceClassification},此时使用下面加载config
if has_remote_code and trust_remote_code:class_ref config_dict[auto_map][AutoConfig] #configuration_chatglm.ChatGLMConfigconfig_class get_class_from_dynamic_module(class_ref, pretrained_model_name_or_path, code_revisioncode_revision, **kwargs)然后会进入get_class_from_dynamic_module方法来加载class,我下面说给函数主要源码来解读首先module_file, class_name class_reference.split(.)将class_ref的configuration_chatglm.ChatGLMConfig字符分成文件和类名 module_fileconfiguration_chatglm, class_nameChatGLMConfig。然后在通过以下代码来调用
# And lastly we get the class inside our newly created module
final_module get_cached_module_file(repo_id, # 文件路径module_file .py, # configuration_chatglm.pycache_dircache_dir,force_downloadforce_download,resume_downloadresume_download,proxiesproxies,tokentoken,revisioncode_revision,local_files_onlylocal_files_only,repo_typerepo_type,
)将final_module变成路径transformers_modules/glm-4v-9b/configuration_chatglm.py
最后在经过get_class_in_module方法来获得config的类class其代码如下
return get_class_in_module(class_name, final_module, force_reloadforce_download)将其结果返回到config_class 中这样就得到了config的方法了。
就是config, kwargs AutoConfig.from_pretrained(*)的返回config内容了该内容实际是继承PretrainedConfig类该方法就是configuration_chatglm.py的文件了该文件如下
from transformers import PretrainedConfigclass ChatGLMConfig(PretrainedConfig):model_type chatglmdef __init__(self,num_layers28,padded_vocab_size65024,hidden_size4096,ffn_hidden_size13696,kv_channels128,num_attention_heads32,seq_length2048,hidden_dropout0.0,classifier_dropoutNone,attention_dropout0.0,layernorm_epsilon1e-5,rmsnormTrue,apply_residual_connection_post_layernormFalse,post_layer_normTrue,add_bias_linearFalse,add_qkv_biasFalse,bias_dropout_fusionTrue,multi_query_attentionFalse,multi_query_group_num1,rope_ratio1,apply_query_key_layer_scalingTrue,attention_softmax_in_fp32True,fp32_residual_connectionFalse,pre_seq_lenNone,prefix_projectionFalse,boi_token_idNone,eoi_token_idNone,**kwargs):self.num_layers num_layersself.vocab_size padded_vocab_sizeself.padded_vocab_size padded_vocab_sizeself.hidden_size hidden_sizeself.ffn_hidden_size ffn_hidden_sizeself.kv_channels kv_channelsself.num_attention_heads num_attention_headsself.seq_length seq_lengthself.hidden_dropout hidden_dropoutself.classifier_dropout classifier_dropoutself.attention_dropout attention_dropoutself.layernorm_epsilon layernorm_epsilonself.rmsnorm rmsnormself.apply_residual_connection_post_layernorm apply_residual_connection_post_layernormself.post_layer_norm post_layer_normself.add_bias_linear add_bias_linearself.add_qkv_bias add_qkv_biasself.bias_dropout_fusion bias_dropout_fusionself.multi_query_attention multi_query_attentionself.multi_query_group_num multi_query_group_numself.rope_ratio rope_ratioself.apply_query_key_layer_scaling apply_query_key_layer_scalingself.attention_softmax_in_fp32 attention_softmax_in_fp32self.fp32_residual_connection fp32_residual_connectionself.pre_seq_len pre_seq_lenself.prefix_projection prefix_projectionself.boi_token_id boi_token_idself.eoi_token_id eoi_token_idsuper().__init__(**kwargs)
4、小节总结
config.json文件内容设置auto_map来实现huggingface调用我们自己构建的ChatGLMConfig类而AutoConfig: “configuration_chatglm.ChatGLMConfig”,的configuration_chatglm为文件py名称ChatGLMConfig为类别名称。同时进一步说明auto_map内容 auto_map: { AutoConfig: configuration_chatglm.ChatGLMConfig, AutoModel: modeling_chatglm.ChatGLMForConditionalGeneration, AutoModelForCausalLM: modeling_chatglm.ChatGLMForConditionalGeneration, AutoModelForSeq2SeqLM: modeling_chatglm.ChatGLMForConditionalGeneration, AutoModelForSequenceClassification: modeling_chatglm.ChatGLMForSequenceClassification },其它方法都可以通过这个方法来调用对应的方法。
