*iv!SSKJr SSKrSSKJrJrJrJrJrJ r SSK J r SSK J r Jr SSKJr SSKrS\R$S'"S S \5r"S S \5r"S S\\5r"SS\5rg)) annotationsN)OptionalListLiteralUnionDictcast)Self)Fieldmodel_validator) BaseSettingsprotected_namespacescv\rSrSr%Sr\"SS9rS\S'\"SSS 9rS \S '\"S S SS9r S\S'\"\ RSS 9r S\S'\"S S SS9r S\S'\"SSS 9rS\S'\"SSS 9rS\S'\"\ R "5SS 9rS\S'\"\ R$"5SS 9rS\S '\"SS!S 9rS"\S#'\"SS$S 9rS \S%'\"\ R,S&S 9rS\S''\"S(S S)S9rS\S*'\"S+S,S-S9rS\S.'\"S+S,S/S9rS\S0'\"\"\R:"5S1-S,5S,S2S9rS\S3'\"\"\R:"5S,5S S4S9rS\S5'\"\ R@S69r!S\S7'\"S8S9S 9r"S:\S;'\"S8SS\Sy'\?"SzS{9S~S|j5r@S}rAg) ModelSettingsz*Model settings used to load a Llama model.z8The path to the model to use for generating completions.) descriptionstrmodelNz9The alias of the model to use for generating completions.defaultr Optional[str] model_aliasrz_The number of layers to put on the GPU. The rest will be on the CPU. Set -1 to move all to GPU.)rgerint n_gpu_layerszThe split mode to use. split_modezMain GPU to use.main_gpuz0Split layers across multiple GPUs in proportion.zOptional[List[float]] tensor_splitFz&Whether to only return the vocabulary.bool vocab_onlyz Use mmap.use_mmapz Use mlock. use_mlockzList of model kv overrides in the format key=type:value where type is one of (bool, int, float). Valid true values are (true, TRUE, 1), otherwise false.zOptional[List[str]] kv_overridesz2comma seperated list of rpc servers for offloading rpc_serverszRandom seed. -1 for random.seedizThe context size.n_ctxizThe batch size to use per eval.n_batchz)The physical batch size used by llama.cppn_ubatchz8The number of threads to use. Use -1 for max cpu threads n_threadszNThe number of threads to use when batch processing. Use -1 for max cpu threadsn_threads_batch)rrope_scaling_typegzRoPE base frequencyfloatrope_freq_basezRoPE frequency scaling factorrope_freq_scalegyarn_ext_factorg?yarn_attn_factorg@@yarn_beta_fastyarn_beta_slow yarn_orig_ctxTz+if true, use experimental mul_mat_q kernels mul_mat_qzWhether to return logits. logits_allzWhether to use embeddings. embeddingz"Whether to offload kqv to the GPU. offload_kqvzWhether to use flash attention. flash_attn@z5Last n tokens to keep for repeat penalty calculation.last_n_tokens_sizezoOptional path to base model, useful if using a quantized base model and you want to apply LoRA to an f16 model. lora_basez*Path to a LoRA file to apply to the model. lora_pathzEnable NUMA support.zUnion[bool, int]numazChat format to use. chat_formatzr?r@rArBrCrDrFrGrHrIrJrKrMrNrOrPr rZ__static_attributes__rr\rYrrs4NE3"'O"K  uL# 00,J &Hc +0F+L'#KJ--/Hd..0 It).o)L%"'H"K ,,:WD#t7JKE3K'HGS'RHcO--/14a8 NIs !O--/3 dOS #==s"#;PQNEQ"!@OU#40OU0#C0e0!$/NE/!#.NE.q)M3)"OItT7RSJSE7STItT"FK#DJ$ K %F I} %@ I} #*D  "')"K&+R&O] SE4*/K*J&QJ /4K/m7<U7$m ',S'm "'^"K(-H( "9FM";FM "GGT r\rc\rSrSr%Sr\"SSS9rS\S'\"SS S9rS \S '\"S S S9r S\S'\"S SS9r S\S'\"S SS9r S\S'\"SSS9r S\S'\"SSS9r S\S'\"SSS9rS\S'Srg )ServerSettingszAServer settings used to configure the FastAPI and Uvicorn server. localhostzListen addressrrhosti@z Listen portrportNzSSL key file for HTTPSr ssl_keyfilezSSL certificate file for HTTPS ssl_certfilezIAPI key for authentication. If set all requests need to be authenticated.api_keyTz=Whether to interrupt requests when a new request is received.r!interrupt_requestsFz;Disable EventSource pings (may be needed for some clients).disable_ping_eventszIThe root path for the server. Useful when running behind a reverse proxy. root_pathr)r^r_r`rarbr rprcrqrrrsrtrurvrxrkrr\rYrmrmsKk7GHD#Hd >D#>!&":"K#("B#L-#_G] %S !&Q!_Isr\rmc\rSrSrSrg)SettingsrN)r^r_r`rarkrr\rYrzrzsr\rzc2\rSrSr%Sr\"/SS9rS\S'Srg) ConfigFileSettingsz#Configuration file format settings.z Model configsrzList[ModelSettings]modelsrN) r^r_r`rarbr rrcrkrr\rYr}r}s-"'"PF Pr\r}) __future__rrTtypingrrrrrr typing_extensionsr pydanticr r pydantic_settingsr rd model_configrrmrzr}rr\rYrsg"=="+*57 01vLvr\> ~} QQr\