*i f
S SK Jr S SKrS SKrS SKrS SKrS SKrS SKrS SKrS SK J r S SK
J r S SK J
r
JrJrJrJrJrJrJrJrJr S SKrS SKJr S SKrS SK J r S SK J r S SK J!r! S SK"J#r# S SK$J%r% SSK&J'r' SS K(J)r)J*r* S
r+S r,S r-S
r.S r/Sr0Sr1Sr2 " S S\5 r3 " S S\45 r5 " S S\*5 r6SS jr7SS jr8\Rr " S S5 5 r: " S S \5 r; " S S \;5 r< SS jr= SS jr> SS! jr? S SS# jjr@ SS$ jrA SS% jrB SS& jrC SS' jrD S SS( jjrE S SS) jjrFSS* jrG SS+ jrH SS, jrI SS- jrJ SS. jrK SS/ jrL SS0 jrM SS1 jrN SS2 jrO SS3 jrPSSS4 jjrQ S SS5 jjrR S SS6 jjrSSS7 jrT\T" S85 SS9 j5 rU\T" S:5 SS; j5 rV\T" S<5 SS= j5 rW\T" S>5 SS? j5 rX\T" S@5 SSA j5 rY\T" SB5 SSC j5 rZ\T" SD5 SSE j5 r[\T" SF5 SSG j5 r\\T" SH5 SSI j5 r]\T" SJ5 SSK j5 r^\T" SL5 SSM j5 r_\T" SN5 SSO j5 r`\T" SP5 SSQ j5 ra\T" SR5 SSS j5 rb\T" ST5 SSU j5 rc\T" SV5 SSW j5 rd\T" SX5 SSY j5 re\T" SZ5 SS[ j5 rf\T" S\5 SS] j5 rg\T" S^5 SS_ j5 rh\T" S`5 SSa j5 ri\T" Sb5 SSc j5 rj\T" Sd5 SSe j5 rk\8" Sf5 SSSSSgShSiSjSkS"/ SSSlSlSmSkS SnSoSSS4 SSp jj5 rl\8" Sq5 \8" Sr5 SSSSSgShSiSjSkS"/ SSSlSlSmSkS SnSoSSS4 SSs jj5 5 rm " St Su5 rn " Sv Sw\n5 ro " Sx Sy\n5 rp " Sz S{\n5 rq " S| S}\n5 rr " S~ S\n5 rs\srt " S S\n5 ru " S S\n5 rv\8" S5 SSSSSgShSiSjSkS"/ SSSlSlSmSkS SnSoSSSSS4 SS jj5 rwg) ) annotationsN)datetime) ExitStack)
AnyDictIteratorListLiteralOptionalTupleUnionProtocolcast) ImmutableSandboxedEnvironment )logger)suppress_stdout_stderr Singletonz{% for message in messages %}{{'<|im_start|>' + message['role'] + '
' + message['content'] + '<|im_end|>' + '
'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
' }}{% endif %}
<|im_end|>a {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token + ' ' }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}a {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}a {% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
' }}{% endif %}c \ rS rSrSrSSSSSSSS/ SSSSSS SSS
S S S S
SSSSSS. SS jjrSrg)LlamaChatCompletionHandler= a Base Protocol for a llama chat completion handler.
Very generic protocol that can be used to implement any chat format.
The only hard requirement is that it must return a ChatCompletion when
stream=False and an iterator of ChatCompletionChunks when stream=True.N皙?ffffff?( F 皙?皙? ?r @皙?) functions
function_calltools tool_choice temperaturetop_ptop_kstreamstopseedresponse_format
max_tokenspresence_penaltyfrequency_penaltyrepeat_penaltymodel
logit_biasmin_p typical_ptfs_z
mirostat_mode mirostat_tau mirostat_etalogits_processorgrammarlogprobs top_logprobsc g N ) selfllamamessagesr$ r% r& r' r( r) r* r+ r, r- r. r/ r0 r1 r2 r3 r4 r5 r6 r7 r8 r9 r: r; r< r= r> kwargss Y/var/www/html/ai-backend/venv/lib/python3.13/site-packages/llama_cpp/llama_chat_format.py__call__#LlamaChatCompletionHandler.__call__D s R rA ) rT returniUnion[llama_types.CreateChatCompletionResponse, Iterator[llama_types.CreateChatCompletionStreamResponse]]__name__
__module__ __qualname____firstlineno____doc__rG __static_attributes__rA rI rF r r = s N IMQU@DLP 02 "
$("%#& # #15 ! !@D04#'&*I) ) A
) F) O) >) J) ) ) ) ) . ) !) "
#) ( ")) * +) , !-) . /) 0 1) 2 /3) 6 7) 8 9) : ;) < =) > ?) @ A) B >C) D .E) F !G) H $I) L
M) ) rI r c \ rS rSrSrg)+LlamaChatCompletionHandlerNotFoundExceptionp rA N)r^ r_ r` ra rc rA rI rF re re p s rI re c Z \ rS rSr% 0 rS\S' S S
S jjrS S jr S S jrSr g)
"LlamaChatCompletionHandlerRegistryt z%Dict[str, LlamaChatCompletionHandler]_chat_handlersc j U( d XR ; a [ SU S35 eX R U' g )NzFormatter with name 'z>' is already registered. Use `overwrite=True` to overwrite it.rj
ValueError)rB name chat_handler overwrites rF register_chat_completion_handlerCLlamaChatCompletionHandlerRegistry.register_chat_completion_handlerw s< T%8%8 8 'v-kl
%1D!rI c Z XR ; a U R U g [ SU S35 e)Nz(No formatter registered under the name 'z'.rl )rB rn s rF unregister_chat_handler:LlamaChatCompletionHandlerRegistry.unregister_chat_handler s1 && &##D) GvR PQ QrI c
U R U nU$ ! [ a4 [ SU S[ U R R 5 5 S35 ef = f)NzInvalid chat handler: z (valid formats: ))rj KeyErrorre listkeys)rB rn ro s rF #get_chat_completion_handler_by_nameFLlamaChatCompletionHandlerRegistry.get_chat_completion_handler_by_name sb ..t4L =(.?TEXEXE]E]E_@`?aabc
s >ArA NF)rn strro r rp rR rn r~ rn r~ r[ r )
r^ r_ r` ra rj __annotations__rq rt r{ rc rA rI rF rh rh t sN <>N9>
1
1 1
1
1R # rI rh c 4 [ 5 R U 5 $ r@ )rh r{ )rn s rF get_chat_completion_handlerr s - / S S rI c ^ SU 4S jjnU$ )Nc < > [ 5 R TU 5 U $ r@ )rh rq )frn s rF decorator3register_chat_completion_handler..decorator s *,MMdTUVrI )r r rA rn r s ` rF rq rq s rI c N \ rS rSr% SrS\S' SrS\S' SrS\S ' S
rS \S ' S
r g)ChatFormatterResponse a
Dataclass that stores completion parameters for a given chat format and
create_chat_completion request.
prompt contains the formatted prompt generated from the chat format and messages.
stop contains the stop token or list of stop tokens to use for the chat format.r~ promptNrS r, z$Optional[llama.StoppingCriteriaList]stopping_criteriaFrR
added_specialrA )
r^ r_ r` ra rb r r, r r rc rA rI rF r r s0 W
K,0D
)0>B;B M4 rI r c . \ rS rSrSr SS jrSrg)
ChatFormatter a Base Protocol for a chat formatter. A chat formatter is a function that
takes a list of messages and returns a chat format response which can be used
to generate a completion. The response can also include a stop token or list
of stop tokens to use for the completion.c g r@ rA )rB rD rE s rF rG ChatFormatter.__call__ s
!$rI rA NrD rK rE r r[ r r] rA rI rF r r s+ 1
$ A$ $
$rI r c \ rS rSr S S
S jjr\S S j5 rSSSSS. S S jjrS
S jrSr g)Jinja2ChatFormatter Nc Xl X l X0l X@l Ub [ U5 OSU l [
[ R " 5 SSS9R U R 5 U l
g)zAA chat formatter that uses jinja2 templates to format the prompt.NT)loader trim_blocks
lstrip_blocks) template eos_token bos_tokenadd_generation_promptsetstop_token_idsr jinja2
BaseLoader from_string _environment)rB r r r r r s rF __init__ Jinja2ChatFormatter.__init__ sd !
""%:"#1#=C 4
:$$&
+dmm
$
rI c J [ R " 5 R U 5 $ r@ )r nowstrftime)r s rF strftime_now Jinja2ChatFormatter.strftime_now s ||~&&q))rI )r$ r% r& r' c
8 ^ SS jnT R R UT R T R UT R UUUUT R
S9
nS n T R b% SU 4S jjn
[ R " U
/5 n [ UT R /U SS9$ )Nc [ U 5 er@ )rm )messages rF raise_exception5Jinja2ChatFormatter.__call__..raise_exception s
W% %rI )
rD r r r r r$ r% r& r' r c ( > U S TR ; $ )N)r )tokenslogitsrB s rF stop_on_last_token8Jinja2ChatFormatter.__call__..stop_on_last_token s bzT%8%888rI T)r r, r r )r r~ )r znpt.NDArray[np.intc]r znpt.NDArray[np.single]r[ rR )
r renderr r r r r rC StoppingCriteriaListr ) rB rD r$ r% r& r' rE r r r r s ` rF rG Jinja2ChatFormatter.__call__ s & "")) nnnn +"&"<"< '# ** *
! *
9,
96L
9
9
!& : :>rI )r r r r r r )TN)
r r~ r r~ r r~ r rR r zOptional[List[int]])r r~ r[ r~ )rD rK r$ rL r% rM r& rN r' rO rE r r[ r )r[ r )
r^ r_ r` ra r staticmethodr rG r rc rA rI rF r r s '+.2
%% % %
$ % ,
%. * * IMQU@DLP)
A)
F )
O )
>
)
J)
)
)
V?rI r c U c g [ U S U S U S 5 VVVVVs/ s H4 u pnUS UUR 5 VVs/ s H u pEUUS S.PM
snnS.PM6 snnnnnS S.$ s snnf s snnnnnf )Nr token_logprobsr> )tokenlogprobbytes)r r r r> )contentrefusal)zipitems)r= r r r> top_token top_logprobs rF )_convert_text_completion_logprobs_to_chatr
s 588JHUeLfhpq iA 5B
5B05< " 3?2D2D2F
! 3G. "+#.!%
3G
!
5B
! !
s A, A&
A, &A, c
SU ; d eSU S - SU S U S SSU S S S
S .[ U S S S 5 U S S S
S./U S S.$ )Nusagechatidchat.completioncreatedr3 r assistantchoicestextroler r=
finish_reasonindexr r= r r objectr r3 r r r )
completions rF _convert_text_completion_to_chatr & s j z$''# i(G$ ' )) 4Q 7 ? FjQZF[\]F^_iFj k!+I!6q!9/!J
G$! rI c # # [ U 5 Hy u pUS:X a SUS - US US SSSS0S S S ./S
.v SUS - US US SSUS S S c
S
US S S 0O0 [ US S S 5 US S S S ./S
.v M{ g 7f)Nr r r r3 r chat.completion.chunkr r r deltar= r r r3 r r r r r r r r= ) enumerater )chunksichunks rF '_convert_text_completion_chunks_to_chatr > s f% 6 uT{* w +1 "# "K" %))-
" 5;&7^ Y'-
!+A.?G &uY'7':6'B I%PYJZ[\J]^hJi j%*9%5a%8 %I
% &s B
B Fc D U( a
U n[ U5 $ U n[ U5 $ r@ )r r )completion_or_chunksr+ r r s rF _convert_completion_to_chatr i s' G[6v>>-A
/
;;rI c
^ U( dl UnSU; d eST - S- US - nSUS - SUS US S S
S T US S S S
.UST US S S S
.S./S.[ US S S 5 SS./US S.nU$ Un SU 4S jjnU" U5 $ )Nr call__0__r r r r r3 r r r r rn argumentsfunctionr typer r r r%
tool_callsr= r r r c 3 D ># SnS nS nS nS nU H nU( a SUS - nUS nUS nST- S- US - nUSUUS S S S
S S S S .S ./S
.v SUS - SUS US S S [ US S S 5 S S TUS S S S.S USTUS S S S.S./S .S ./S
.v SnM Uc eSUS - SUS US S S [ US S S 5 S S TUS S S S.S USTUS S S S.S./S .S ./S
.v M Ub Ub Ub USUUS SS S S S S S .S ./S
.v g g g g 7f)NTr r r r3 r r r r r r r r r= r r r r r3 r r r= r r r r r r r Fr r )r firstid_r r3 tool_idr tool_names rF #_stream_response_to_function_streamQ_convert_completion_to_chat_function.._stream_response_to_function_stream sk ECGEG 5;.C #I .G !'NE - 9C ?%+ MG !"9#*!& *+15,0,7/35926 *"
$ ( %uT{ 2"9#(#3!&w *+15,UV[\eVfghVijtVu,v,0/3095:95Ea5H5P6& 67294>8A=B9=Ma=P4:>29.
)* 3&*" $ ! !D "E *** 5;. 5 $Y / "7^ &'-1(QRWXaRbcdRefpRq(r(,+/,516y1A!1DV1L2" 23.50:4=9>y9I!9LV9T5* %&
/"& } ~ 7#65;L 5 & " &'-9(,(,+/15.2 &
q)A&)I* '.(2,51;I1Fq1I&1Q-" ' $ !J*U^J_`aJbcmJn o%1+2 (= E
@ G[{ H{
E{ z 36::rI c ^ S S S S SSSSSS/ S S S SSSSS S
S S S S S S S S . SU 4S
jjjnU$ )Nr r r r r! Fr r r r" r# r$ r% r& r' r( r) r* r5 r6 r+ r, r- r. r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< r4 r= r> c >^( T)" UUUUUS9n U R " U R R S5 U R ( + SS9n U R b\ U c / O[ U [ 5 ( a U /OU n [ U R [ 5 ( a U R O U R /n U U - n S n!U R b U R n!Ub US S:X a [ XR S9nUb U V"s/ s H n"SU"S .PM
nn"UbH [ U[ 5 ( a US
:X d US :X a Un[ U[ 5 ( a S U; a
SS US 0S .nS n#Ub [ U[ 5 ( ax Ubu US S m([ U(4S
jU 5 S 5 n#U#c [ ST( S35 eU#S S n$ [ R R! [" R$ " U$5 U R S9nU R2 " S*0 SU _SU_SU_SU_SU _SU
_SU( a U OS _SU _SU _SU
_S U_S U_S U_S U_S U_S!U_S"U_S#U_S$U_S%U_S&U!_S'U_S(U_6n&U#b U#S S n'[5 U'U&U 5 $ [7 U&U S)9$ s sn"f ! [& ar n%U R ( a! [) [
U%5 [* R, S9 [ R R/ [ R0 U R S9n S n%A%NS n%A%ff = f)+N)rD r$ r% r& r' utf-8T)add_bosspecialr json_objectverboser r r noneautorn c 3 B ># U H oS S T:X d M Uv M g7fr rn NrA .0trn s rF ]chat_formatter_to_chat_completion_handler..chat_completion_handler.. ! KEqz]6-Bd-JE
Tool choice '' not found in tools.
parametersfiler r( r) r* r5 r6 r= r+ r, r- r/ r0 r1 r2 r7 r8 r9 r: r3 r; r r< r4 r+ rA ) tokenizer encoder r,
isinstancer~ ry r _grammar_for_response_formatr
dictnextrm
llama_grammar LlamaGrammarfrom_json_schemajsondumps Exceptionprintsysstderrr JSON_GBNFcreate_completionr r )*rC rD r$ r% r& r' r( r) r* r5 r6 r+ r, r- r. r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< r4 r= r> rE resultr rstopr r toolschemaer r rn chat_formatters* @rF chat_completion_handlerJchat_formatter_to_chat_completion_handler..chat_completion_handler. s L '#
MM ) ,,,
;; " 2Zc5J5JD6PTD#-fkk4#@#@FKKv{{mE% rT r[ r\ rA )r6 r7 s ` rF r r + s IMQU@DLP 02 "
$("%#& # ! ! #@D0415#'&*AMPMP AMP F MP
O MP >
MP JMP MP MP MP MP MP MP . MP MP
!MP& "'MP( )MP* !+MP, -MP. /MP0 1MP2 3MP4 5MP6 7MP8 >9MP: .;MP< /=MP> !?MP@ $AMPD
EMP MP^ #"rI c R ^ SSK Jn UR U 5 m SU4S jjnU$ )Nr
AutoTokenizerc > STl TR U SS9n[ U[ 5 ( d e[ UTR
SS9$ )NFr Tr r, r )use_default_system_promptapply_chat_templater" r~ r r )rD rE r tokenizers rF format_autotokenizer@hf_autotokenizer_to_chat_formatter..format_autotokenizer sO /4 + 33Hu3M&#&&&&$ 3 34
rI r ) transformersr< from_pretrained) pretrained_model_name_or_pathr< rC rB s @rF "hf_autotokenizer_to_chat_formatterrH s@ + --.KLI
@
rI c . [ U 5 n[ U5 $ r@ )rH r )rG r6 s rF +hf_autotokenizer_to_chat_completion_handlerrJ s 88UVN 4^ DDrI c z ^^^^ [ U [ 5 ( d eSU ; d e[ U S [ 5 ( d eU S nSU ; d e[ U S [ 5 ( d eU S mSU ; d e[ U S [ 5 ( d eU S m[ SSS9R U5 m SUUUU4S jjnU$ )N
chat_templater r Tr r c > T( a / U Q[ R " SSS9Pn TR U TTS9n[ UTT/SS9$ )Nr r )rD r r Tr? ) llama_types%ChatCompletionRequestAssistantMessager r )rD rE r r r envr s rF format_tokenizer_configFhf_tokenizer_config_to_chat_formatter..format_tokenizer_config sh !AA$bH
%I 6d
rI r )r" r$ r~ r r )tokenizer_configr rL rS r rR r s ` @@@rF %hf_tokenizer_config_to_chat_formatterrV s & - -- - . .. . &7 = == =$_5M * ** * &{3S 9 99 9 -I * ** * &{3S 9 99 9 -I
' k-
@
, #"rI c * [ XS9n[ U5 $ )N)r )rV r )rU r r6 s rF .hf_tokenizer_config_to_chat_completion_handlerrX s ;N 5^ DDrI c SU ; a g U S [ :X a gU S [ :X d
U S [ :X a gU S [ :X a gg )Nztokenizer.chat_templatechatmlmistral-instructllama-3)CHATML_CHAT_TEMPLATE MISTRAL_INSTRUCT_CHAT_TEMPLATE MIXTRAL_INSTRUCT_CHAT_TEMPLATE LLAMA3_INSTRUCT_CHAT_TEMPLATE)metadatas rF $guess_chat_format_from_gguf_metadatarb sW 0)*.BB *+/MM - .2P P!)*.KK rI c H U H nUS S:X d M US =( d Ss $ g)z Get the first system message.r systemr rO rA )rD r s rF _get_system_messagere 1 s/ 6?h &9%+ + rI c / nU HC nUS nXA; d M [ US [ 5 ( a US OSnUR X U45 ME U$ )zMap the message roles.r r N)r" r~ append)rD role_mapoutputr r r s rF
_map_rolesrj ; s^
/1Fv &01CS&I&I "t
MM8>73 4
MrI c X#/nX- n[ U5 HP u nu pxU ( a US:X a U=( d Sn XYXFS- - -
nM. U( a XWU- S- XFS- - -
nMI XWS- -
nMR U$ )z(Format the prompt with the llama2 style.r rO r )
system_messagerD sepsep2sepsretr r r ms
rF _format_llama2ru K s
;D
C ' 1?D a1f
2A tE{? "C
'>C'$1u+5 5C #: C 2 JrI c ^ X- nU H" u pEU( a X4S- U- U- -
nM X4S- -
nM$ U$ )z2Format the prompt with the add-colon-single style.: :rA ro rD rp rs r r s rF _format_add_colon_singlerz \ sD
C!
$;(3. .C #: C "
JrI c X#/nXS - n[ U5 H* u nu pxU( a XWS- U- XFS- - -
nM# XWS- -
nM, U$ )z/Format the prompt with the add-colon-two style.r rw rl rx rn ) ro rD rp rq rr rs r r r s rF _format_add_colon_twor| i s_
;D
7
"C ' 1?D $;(4A;6 6C #: C 2
JrI c N U nU H u pEU( a X4U- U- -
nM X4-
nM U$ )z1Format the prompt with the no-colon-single style.rA ry s rF _format_no_colon_singler~ w s7 C!
'>C' 'C KC "
JrI c ^ X- nU H" u pEU( a X4S- U- U- -
nM X4S- -
nM$ U$ )z8Format the prompt with the add-colon-space-single style.rw rA ry s rF _format_add_colon_space_singler sD
C!
$;(3. .C $; C "
JrI c z U S:X a SOX- S- nU H% u pEU( a X4S- U- U- S- -
nM X4S- -
nM' U$ )z(Format the prompt with the chatml style.rO
rA ry s rF _format_chatmlr sW "$".*>*EC!
$;(3.5 5C $; C "
JrI c j SnU ( a X0-
nU H u pEU( a X4S- S- U- -
nM X4-
nM! U$ )z*Format the prompt with the chatglm3 style.rO r rm rA ry s rF _format_chatglm3r sG
C !
$;$w. .C KC "
JrI c Z [ R R [ R U S9$ Nr )r& r' r r/ r s rF _grammar_for_jsonr s+ % % 1 1 2 rI c [ R R XS9$ ! [ a nU( a [ US9s S nA$ UeS nAff = fr )r& r' r( r+ r )r4 r
fallback_to_jsonr5 s rF _grammar_for_json_schemar sD ))::6:SS $W5 5G s
AA AA Ac v U S S:w a g SU ; a! [ [ R " U S 5 US9$ [ US9$ )Nr r r4 r )r r) r* r )r. r
s rF r# r# sI v -/?"' JJx0 17
!11rI c ^ SU 4S jjnU$ )Nc R > [ U 5 n[ 5 R TU5 U $ r@ )r rh rq )r r7 rn s rF r 'register_chat_format..decorator s, "KA"N *,MM )
rI )r r rA r s ` rF register_chat_formatr s rI zllama-2c Sn[ SSS9n[ X5 n[ U 5 nU( a UR US9n[ XTSS5 S- n[ US9$ ) Nz([INST] <>
{system_message}
<>z [INST]z[/INST]userr ro rm r r )r$ rj re formatru r )rD rE _system_template_roles _messagesro _prompts rF
format_llama2r s_
D
{i
8F8,I(2N)000O ^VDyPG 00rI r\ c [ SSSS9nSn[ X5 nUR US S 45 [ SXC5 n[ XSS9$ ) Nz,<|start_header_id|>system<|end_header_id|>
z*<|start_header_id|>user<|end_header_id|>
z/<|start_header_id|>assistant<|end_header_id|>
)rd r r z
<|eot_id|>r rO r r, )r$ rj rg r~ r )rD rE r _sepr r s rF
format_llama3r sW
?
;EF
D8,I
f[)401%b):G ;;rI alpacac t [ SSS9nSnSn[ U 5 n[ X5 n[ XVX45 n[ US9$ )Nz### Instruction ### Responser
r r )r$ re rj r| r )rD rE r r _sep2ro r r s rF
format_alpacar sF
(N
CF D E(2N8,I#NtKG 00rI qwenc [ SSS9n[ U 5 =( d SnSnUR US9n[ X5 nUR US S 45 Sn[ X5U5 nS n[
XxS
9$ ) N<|im_start|>user<|im_start|>assistantr You are a helpful assistant.#<|im_start|>system
{system_message}r r r z
<|endoftext|>r )r$ re r rj rg r r ) rD rE r ro system_templater r r r s rF format_qwenr s}
)5L
MF(2T6TN+JN8,I
f[)401 D ^=G E <>
{system_message}
<>
<|prompter|>
<|assistant|>r r r r r r$ re r rj rg r~ r rD rE r r r ro r r s rF format_oasst_llamar / sr
H
~
AF D(2N%,,N,KN8,I
f[)401%nFG 00rI z
baichuan-2c Sn[ SSS9nSn[ U 5 nUR US9n[ X5 nUR US S 45 [ XVU5 n[
US9$ ) N{system_message}zzr rO r r r r r s rF format_baichuan2r ? s
*
'3C
DF
D(2N%,,N,KN8,I
f[)401%nFG 00rI baichuanc Sn[ SSS9nSn[ U 5 nUR US9n[ X5 nUR US S 45 [ XVU5 n[
US9$ ) Nr zzr rO r r r r r s rF format_baichuanr O r rI openbuddyc Sn[ SSS9nSnUn[ X5 nUR US S 45 [ XVU5 n[ US9$ )Na You are a helpful, respectful and honest INTP-T AI Assistant named Buddy. You are talking to a human User.
Always answer as helpfully and logically as possible, while being safe. Your answers should not include any harmful, political, religious, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
You can speak fluently in many languages, for example: English, Chinese.
You cannot access the internet, but you have vast knowledge, cutoff: 2021-09.
You are trained by OpenBuddy team, (https://openbuddy.ai, https://github.com/OpenBuddy/OpenBuddy), you are based on LLaMA and Falcon transformers model, not related to GPT or OpenAI.
User Assistantr r r r r$ rj rg rz r )rD rE r r r ro r r s rF format_openbuddyr _ sZ
O v
5F D$N8,I
f[)401&~$GG 00rI zredpajama-incitec [ U 5 n[ SSS9nSnSnUn[ X5 nUR US S 45 [ XgU5 n[ XS9$ )Nzzr r r r )re r$ rj rg rz r ) rD rE r r r _stopro r r s rF format_redpajama_inciter u sb
*(3O
yG
4F D E$N8,I
f[)401&~$GG <F D E$N8,I
f[)401&~$GG <
?F D5O8,I
f[)401&4HG 00rI z open-orcac SnSnSnSnSnUR US9n[ U [ [ XD5 5 5 nUR US S 45 [ X7U5 n[
XS9$ ) Nr a You are a helpful assistant. Please answer truthfully and write out your thinking step by step to be sure you get the right answer. If you make a mistake or encounter an error in your thinking, say so out loud and attempt to correct it. If you don't know or aren't sure about something, say so clearly. You will act as a professional logician, mathematician, and physicist. You will also act as the most appropriate type of expert to answer any particular question or solve the relevant problem; state which expert type your are, if so. Also think of any particular named expert that would be ideal to answer the relevant question or solve the relevant problem; name and act as them, if appropriate.)r r z<|end_of_turn|>
r r r r )r rj r$ r rg r r ) rD rE r ro rolesrp stop_strr r s rF format_open_orcar s|
)O B
"E
CH$++>+JN8T#e*;%<=I
eAh%&,^LG ??rI mistrallitec [ SSS9nSnSn[ U 5 nUR US9n[ X5 nUR US S 45 [ XVU5 n[
US9$ ) Nr z
<|assistant|>r rm z <|system|>{system_message}r r r r )rD rE r r r ro r r s rF format_mistralliter sq
~1F
GF D:O(2N$++>+JN8,I
f[)401%nFG 00rI zephyrc Sn[ U 5 nUR US9n[ SSS9nSn[ X5 nUR US S 45 [ X6U5 n[
XuS9$ ) N<|system|>
{system_message}r z <|user|>
z<|assistant|>
r r r r re r r$ rj rg r r rD rE r ro r r r r s rF
format_zephyrr sq
O(2N$++>+JN
|/@
AF D8,I
f[)401 ^=G ;;rI pygmalionc Sn[ U 5 nUR US9n[ SSS9nSn[ X5 nUR US S 45 [ X6U5 n[
XuS9$ ) Nz<|system|>{system_message}r <|user|>z <|model|>r r r r r r s rF format_pygmalionr sp
7O(2N$++>+JN
z[
9F D8,I
f[)401 ^=G ;;rI rZ c Sn[ U 5 nUR US9n[ SSS9nSn[ X5 nUR US S 45 [ X6U5 n[
XuS9$ ) Nr r r r r r r r r r s rF
format_chatmlr sr
O(2N$++>+JN
)5L
MF D8,I
f[)401 ^=G ;;rI r[ c SnUnSnU HX nUS S:X a+ US b% [ US [ 5 ( a
USUS - -
nM7 US S:X d MB US c MJ USUS - U- -
nMZ US-
n[ XCS 9$ )
Nr rO r r r z[INST] r z [/INST]r )r" r~ r )rD rE eosr, r r s rF format_mistral_instructr s
C D
F FOv % ".79-s33 i')"44 4F
V_
+ 0B0N j79#55; ;F jF ::rI chatglm3c Sn[ U 5 nUR US9n[ SSS9nSn[ X5 nUR US S 45 [ X6U5 n[
XuS9$ ) Nr r r r r r r r )re r r$ rj rg r r r s rF format_chatglm3r ) sp
O(2N$++>+JN
z_
=F D8,I
f[)401 ~$?G ;;rI openchatc Sn[ U 5 nUR US9n[ SSS9nSn[ X5 nUR US S 45 [ X6U5 n[
XuS9$ ) Nz {system_message}<|end_of_turn|>r zGPT4 Correct User: z'<|end_of_turn|>GPT4 Correct Assistant: r z<|end_of_turn|>r r r r s rF format_openchatr : sv
8O(2N$++>+JN
".WF D8,I
f[)401 ^=G ;;rI saigac Sn[ SSSS9n[ X5 nSnU H( u pgU( a XRR XgS9-
nM USU S 3-
nM* US
-
n[ UR 5 S 9$ ) Nz{role}
{content}r botrd )r r rd rO r r r zbotr )r$ rj r r strip)rD rE _message_templater r r r r s rF format_saigar N s{
3
v5
:F8,IG"
//T/K KG TF"~ %G # xG
88rI gemmac [ U 5 nUS:w a [ R " S5 [ SSS9nSn[ X5 nUR US S 45 [
SXTS9n[ XdS 9$ )
NrO zB`role='system'` messages are not allowed on Google's Gemma models.zuser
zmodel
r z
r )ro rD rp r )re r debugr$ rj rg r~ r )rD rE ro r r r r s rF format_gemmar d ss
)2N P
.:R
SF D8,I
f[)401%R)VG ;;rI functionaryr r r r r! r r r" r# c ~ ^%^&^'^( Sm% SS?U'U(4S jjjm& S@ SAU%U&4S jjjnUb U Vs/ s H nUS S:X d M US PM nnUb [ U[ 5 ( a UOUS nU" XU5 n Ucj Ub [ U5 S:X aX U R " SB0 S U S
- _S U_S U_S
U_SU _SU
_SU _SSS/_SU_SU_SU_SU_SU_SU_SU_SU_S U_S U_S U_6n [ U U S 9$ Ub [ U[ 5 ( aE US :X a? S!n U R " U U S"S#9n U S$ S S% n U R S&5 S' S S' nU U - U - n OM[ U[ 5 ( a US(:w a U S
- n O,[ U[ 5 ( a U S)US* S
3- n US* nOU S
- n S n!U=( d / H n"U"S* U:X d M U"S+ n! O U=( d / H$ nUS S:X d M US S* U:X d M US S+ n! O U!b [ U R S,9 [ R " [ R " U!5 5 n#[ R R [ R " [ R " U!5 5 U R S-9n[ U#5 S S S 5 OR[ U R S,9 [ R R [ R" U R S-9nS S S 5 U R " SB0 S U _SSS/_SS"_S U_SU_S U_S U_S
U_SU _SU
_SU_SU_SU_SU_SU_SU_SU_S U_S U_6n S/U ; d e[ U[ 5 ( d eU S"L d eU R ( a [ U 5 [ U S$ S S% 5 [$ R& " S0U S1 - S2U S3 U S SS4S UU S$ S S% S5.USUU S$ S S% S5.S6./S7.[) U S$ S S8 5 S9S:./U S/ S;9$ s snf ! , ( d f GN= f! [ a n$U R ( a [ S.5 [ U$5 [ U R S,9 [ R R [ R" U R S-9nS S S 5 S n$A$GM ! , ( d f S n$A$GN= fS n$A$ff = f! , ( d f GN= f)CNA chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant calls functions with appropriate input when necessaryc > SU- nSU ; a U S R S5 S nU$ U R S5 S:X a$ U R S0 5 nT
" XQS- U5 nS U S
3$ U R S5 S :X aP U R S 0 5 nS
nUR 5 H u pT
" XS- U5 n UU SU SU S3-
nM! XS- -
nU$ SU ; a) SR U S V s/ s H n SU S3PM sn 5 $ U R SS5 $ s sn f N z$ref/r r arrayr r zArray<>r
propertiesz{
rw ,
}enum | "anysplitgetr joinparam indent_level shared_defsindentref_namer item_typer
nested_schemanested_param_name nested_paramnested_param_type
enum_valuegenerate_type_definitions rF r :functionary_chat_handler..generate_type_definition R $ U? V}**3/H O
YYv
'
)IIgr*E0q8H+VII;a( (
YYv
(
* <4J !M3=3C3C3E/!$< "2K%! hb!2 326G5HL
4F
c\ )M
u_::5= Q=Z:,a 0= QR R 99VU+ + R Dc
> SU- nSnU R 5 Hh u pEX2 SU S3-
nUR S5 S:X a
UT" XQU 5 -
nO2SU; a, USR US Vs/ s H nS U S 3PM sn5 -
nUS
-
nMj U$ s snf Nr rO type z = r r r r r z;
r r
r r r r shared_definitionsdef_namedef_propertiesr r s rF generate_shared_definitions=functionary_chat_handler..generate_shared_definitions $ (3(9(9(; $H HE(3"? ? !!&)X5"&>"+' " >)"ejj9G9OP9O:q A&9OP' "
% ' )< "! Q #B c Z > SnUSU S3-
n0 nU H6 nUR S0 5 nUR UR S0 5 5 M8 UT" US5 -
nU H nUS nUR SS 5 nUR S0 5 nUR S
/ 5 nUS U S 3-
nUS
U S3-
nUR S0 5 R 5 HA u pU
R SS 5 n T" U
SU5 n X; a S OSn
USU S 3-
nUSU U
SU S3-
nMC US-
nM USR U5 -
nU$ )NH// Supported function definitions that should be called when necessary.
namespace {
r $defsr rn descriptionrO requiredz // r z type = (_: {
r rl ?z // z rw r z }) => any;
z}} // namespace {}
r
updater r r$ namespacer4 r$ r r
function_namer0 required_params
param_namer param_description
param_typeoptional_indicatorr' r s rF generate_schema_from_functions@functionary_chat_handler..generate_schema_from_functions s W Jyk11 !H! lB7J % %jnnWb&A B " -.@!DD!H $V ,M",,}b9K! lB7J (nnZ / nUb* UR [ R " ST " U5 S95 UbI UR [ R " ST " U Vs/ s H nUS S:X d M US PM sn5 S95 UR [ R " STS95 U HE nUS S:X a SU; a SUS 3US' SU; a SUS S 3US S' UR U5 MG UR [ R " S S S95 S S
jnS R U Vs/ s H
ov" U5 PM sn5 $ s snf s snf )
Nrd r r r r rn
functions.r% r c p U S S:X a SU S S3$ U S S:X a SU ; a SU S S U S S3$ U S S:X a S
U ; a SU S
S S U S
S S3$ U S S :X a U S b SU S
S U S S3$ SU S
S3$ U S S:X a U S c gSU S S3$ U S S:X a U S b! S
U ; a SU S SU S
S S U S
S S3$ S
U ; a SU S
S S U S
S S3$ SU ; a1 [ U S 5 S: a U S H nSUS S US S S3s $ g U S c gSU S S3$ [ SU S 35 e)Nr rd zsystem:
r r r rn zfunction name=:
r% r r3 tool_call_idr zuser:
z
user:
z
r assistant:
z
assistant to=z
assistant to=r r r zUnsupported role: )lenrm )msg tool_calls rF message_to_strXfunctionary_chat_handler..prepare_messages_for_inference..message_to_str s> 6{h&"3y>"2"55V
*v}'F }CI7GrJJV
*#/E'O(- +C,?+@C NCSSUVV +C,?+@CCV&y>). (Y (8??V +y>-/S2H )#i.)9 _I]^dIeHffijmn}j~ @K kL jM MS T T$+ *3+?+G*HCP_L`alLmKnntuu!S(S\1B-Ca-G%($& "/y.?s9ZCXYdCeBffl mm& ^+& )#i.)9<< #5c&k]!CDDrI rO )rG z(llama_types.ChatCompletionRequestMessage)rg rP "ChatCompletionRequestSystemMessagerQ r )
rD r$ r& all_messagesr3 r rI rG SYSTEM_MESSAGEr> s
rF prepare_messages_for_inference@functionary_chat_handler..prepare_messages_for_inference s
HJ >>!+I)+T
>>! : ).(- #F|z 9 -D ,(-
: : ~
Gv*,71B$.wv.?"@ ') ! !9&!A BC (
( = = $
! EF ww|D|s+|DEEGF Es
D<' D<$Er r r r rC r( r) r* r5 r6 r+ r, zuser:r r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< r r r F)r r, r+ r r .r r z to=functions.rn r disabler MFailed to parse function body as JSON schema, falling back to default grammarr r r r r r r r r r= r r r r z Dict[str, llama_types.JsonType]r rQ r[ r~ r rQ r[ r~ r$ r[ r~ NN)rD rK r$ 3Optional[List[llama_types.ChatCompletionFunctions]]r& rN rA )r" r~ rF r0 r r r$ r r
r& json_schema_to_gbnfr) r* r' r r, r+ r/ rP CreateChatCompletionResponser ))rC rD r$ r% r& r' r( r) r* r5 r6 r+ r, r. r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< rE rN r3 r completion_or_completion_chunksr completion_text
new_prompt
function_bodyr grammar_textr5 rM r> r' r s) @@@@rF functionary_chat_handlerra y s< : zN ,. ,>A , ,B"" H JN@DUF@UFFUF >UF UFn
27V%$4<:;U%T*%% V %k377K[=T ,H
GF )"3s9~7J*/*A*A +
E>+
#+
+
+
+
+
+
6"+
"+
.+
0+
*+
+
( +
& +
&!+
" #+
$ .%+
& '+
'* ++JSYZZ =#&&=F+B-2-D-DU.
%Y/26:'--c226s;
o-4
M3 ' 'MV,Ce^
M4 ( (}V/D.ES II
%f-
e^
M OO F } , $\ 2M $ <: %$z*:6*Bm*S ,\ :M
'
> , @ @JJ}- (44@@!55djj 6OP !MM A l# ?>* $EMM
:#00<< '' = G ;
*/)@)@ **v
* * *
*
* * * * * ** ,* &* * $ * "!*" "#*$ %*& *'*J, j mS ) )) ) U?? }}
j
j#A&v./ 3 3 Jt$ $ 9%! ' # -%/ %:1%=f%E& #0$.(5-7 -B1-Ef-M) # $ FjQZF[\]F^_iFj k!-+
2 != s W@ ?>
}}c a'
>'44@@!++ !MM A ?>>>>
;
:sm
O. O.7P
BO3P ,7R-3
P=P P
R*:R% 7R R%
R" R%"R%%R*-
R<zfunctionary-v1zfunctionary-v2c ^ ^^^^^^ ^
^ ^^^^^^^^^^^4^5^6^7^8^9^:^;^<^=^> Sm8T R n[ US5 ( d S5 eSSKJn SUR R
; a Sm>Sn S n S
n S n Sn S n!O
S
m>Sm6Sm5Sm7Sm4 SVU<4S jjmTTU5 n$US:X d Tb [ T5 S:X a T>S:X a W n OT7n U$S-
n$T R " S\0 SU$_S T_S T_S T_S T _S T
_S!T _S"U _S#T_S$T_S%T_S&T_S'T_S(T_S)T_S*T_S+T_S,T_S-U_6n%T S.L a" U%S/ S S0 R 5 U%S/ S S0' [ U%T S19$ UU U4S2 jm=UU UUU UUUUUUU UUUUU
4S3 jm9S4n&/ / n(n'Sn)U4U5U6U7U9U=U>4S5 jn*T S.La U*" TTUU$S69$ T>S:X Gai [
U[ 5 ( a US:X a S7W /n+OI[
U[ 5 ( a. U$W US8 S93-
n$W!n+US8 nU'R U5 T=" U5 nOU$n$S7W /n+T9" U$U+US:9n,U,S/ S S0 n-U)U,S; S< -
n)W U$; a U U-; a U)U,S; S<' [ U,T S19$ U U$; a U U-; a U$U-R U S=3U 5 S7- -
n$U'R U-R U 5 S> S S> R! 5 5 T=" U'S> 5 nT9" U$W!US:9n,U)U,S; S< -
n)U(R U,S/ S S0 R! 5 5 GOU(R U-R! 5 5 GO[
U[ 5 ( an U$US8 S7T4 3-
n$US8 nU'R U5 T=" U5 nT7T5/n+T9" U$U+US:9n,U,S/ S S0 n-U)U,S; S< -
n)U(R U-R! 5 5 GO[
U[ 5 ( Ga US:X Ga S nT4n+T9" U$U+US:9n,U,S/ S S0 n-U)U,S; S< -
n)U-R! 5 n.U.S?:X a U$S-
n$O1U-R! 5 nU$U S@3-
n$U'R U5 T=" U5 nT6T7/n+T9" U$U+US:9n,U,S/ S S0 n-U)U,S; S< -
n)U.S?:X a U-R# SA5 ( a U&U-S [ SA5 * -
n&U-R# SB5 ( a U&U-[ SB5 * -
n&OU&U--
n&U&R 5 n&SCU-; d SDU-; a U-R# SA5 ( a U-S [ SA5 * R! 5 n/ODU-R# SB5 ( a U-[ SB5 * R! 5 n/OU-R! 5 n/U$U/ SE3-
n$OmOoU(R U-R! 5 5 U$U-R! 5 -
n$S nT9" U$U+US:9n,U)U,S; S< -
n)SCU,S/ S S0 ; d SDU,S/ S S0 ; a U$SE-
n$OOGM S;W,; d e[ U'5 [ U(5 :X d e/ n0[% U'U(5 H} u nn1U0R SFS4R' [) SG5 V2s/ s H8 n2[* R, " [. R0 [. R2 - 5 PM: sn25 - SUU1SH.SI.5 M 0 n3[ U05 S: a# Tb U0U3SJ' OU0S S S8 U0S S SK SH.U3SL' U)U,S; S<' [4 R6 " SMU,SN - SOU,SP U,S+ S[9 U,S/ S SQ 5 SRU&S4:X a S OU&SS.U3E[ U05 S: a SJOS"ST./U,S; SU9$ s sn#f s sn2f )]Nr hf_tokenizerznPlease provide a valid hf_tokenizer_path from https://huggingface.co/meetkai when initializing the Llama classr r; z<|START_OF_FUNCTION_CALL|>v1z<|END_OF_SYSTEM|>z<|END_OF_USER|>z<|END_OF_ASSISTANT|>z<|END_OF_FUNCTION_RESULT|>z<|END_OF_FUNCTION_CALL|>v2z
<|recipient|>z<|from|>z<|stop|>z <|content|>c > SU- nSU ; a U S R S5 S nU$ U R S5 S:X a$ U R S0 5 nT
" XQS- U5 nS U S
3$ U R S5 S :X aP U R S 0 5 nS
nUR 5 H u pT
" XS- U5 n UU SU SU S3-
nM! XS- -
nU$ SU ; a) SR U S V s/ s H n SU S3PM sn 5 $ U R SS5 $ s sn f r r r s rF r @functionary_v1_v2_chat_handler..generate_type_definition r r c
> SU- nSnU R 5 Hh u pEX2 SU S3-
nUR S5 S:X a
UT" XQU 5 -
nO2SU; a, USR US Vs/ s H nS U S 3PM sn5 -
nUS
-
nMj U$ s snf r r" r# s rF r' Cfunctionary_v1_v2_chat_handler..generate_shared_definitions5 r) r* c V > SnUSU S3-
n0 nU H6 nUR S0 5 nUR UR S0 5 5 M8 UT" US5 -
nU H nUS nUR SS 5 nUR S0 5 nUR S
/ 5 nUS U S 3-
nUS
U S3-
nUR S0 5 R 5 H? u pU
R SS 5 n T" U
SU5 n X; a S OSn
US U S 3-
nX) U
SU S3-
nMA US-
nM USR U5 -
nU$ )Nr, r- r. r r/ r rn r0 rO r1 z// r r! r2 r rl r3 rw r z }) => any;
z}} // namespace {}r4 r6 s rF r> Ffunctionary_v1_v2_chat_handler..generate_schema_from_functionsF s W Jyk11 !H! lB7J % %jnnWb&A B " -.@!DD!H $V ,M",,}b9K! lB7J (nnZ / nUS:X a+ UR [ R " ST " / 5 S95 OUb+ UR [ R " ST " U5 S95 ORUbO US:w aI UR [ R " ST " U Vs/ s H nUS S:X d M US PM sn5 S95 UR [ R " ST
S95 U HE nUS S:X a SU; a SUS 3US' S U; a SUS S 3US S' UR U5 MG US
:X a S n OS n UR R US
S9U - $ s snf )Nr rd r r r r rn rA r% rd rE z <|from|>assistant
<|recipient|>Fr> )rg rP rK rc rA ) rD rB versionr$ r& r' rL r3 r suffixrM r> s rF rN Ffunctionary_v1_v2_chat_handler..prepare_messages_for_inferenceh s HJ & >>!+I"+M
$ ## BB %/Mi/X
"{f'< ## BB % > -2 ,1D#'<:#= !1Z 0,1 ! : : ~
Gv*,71B$.wv.?"@ ') ! !9&!A BC (
( d?#F7F
" " 6 6|e 6 T
? s
E E r r r all
<|content|>r r( r) r* r5 r6 r+ r, r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< Fr r r c Z > S nT=( d / H nUS U :X d M US n O T =( d / H$ nUS S:X d M US S U :X d M US S n O [ TR S9 [ R " [ R
" U5 5 n[ R R [ R " [ R
" U5 5 5 n[ U5 S S S 5 U$ ! , ( d f W$ = f! [ a nTR ( a [ S5 [ U5 [ TR S9 [ R R [ R TR S9nS S S 5 S nAU$ ! , ( d f S nAW$ = fS nAff = f)Nrn r r r rQ rS r ) r r
r& rZ r) r* r' r r, r+ r/ )
r% r_ r r3 r` r< r5 r$ rC r& s
rF get_grammar3functionary_v1_v2_chat_handler..get_grammar sy
!RH =0 ( 6
( KRKDF|z)d:.>v.F-.W $Z 0 >
'
> , @ @JJ}- (44@@!55djj 6OP l# ?& ' ?>& }}c a'
>'44@@!++U]] A ?>
?>>
sU D -A=C4*D 4
D>D D
F*:F%
7FF%
F" F%"F%%F*c > [ [ R TR " S0 SU _ST_ST_ST_ST_ST_ST_SU_S T_S
T
_S T_S T_S
T_ST
_ST _ST _ST _ST_SU_65 nU$ )Nr r( r) r* r5 r6 r+ r, r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< rA )r rP
Completionr0 )r r, r< r r1 rC r; r/ r5 r: r8 r9 r3 r0 r2 r+ r( r7 r* r) r6 s rF r0 9functionary_v1_v2_chat_handler..create_completion s " " # #
'
$
&
"2
#4
.
,
*
*!
" #
$ "2%
& '
2 rI rO c 3 ># T S:X d S5 eSu pE[ U[ 5 ( Ga X2S ST 3-
nT" US 5 nTT/nSR [ S5 Vs/ s H8 n[ R
" [ R [ R - 5 PM: sn5 n T" X7US9n
Sn S n U
GH n
U ( aS U b S
S S U - S
US SS.S./0nO
SUS SS.0n[ R " SU
S - SU
S U
S S S S S S.UES./S9v Sn U b( S
S S U - S
S U
S S S R 5 S.S./0nO SS U
S S S R 5 S.0n[ U
S S S R 5 5 S : d M [ R " SU
S - SU
S U
S S [ U
S S S 5 S S S.UES./S9v GM [ R " SW
S - SU
S U
S S U b S
OSS S S S S S .S ./S9v g [ U[ 5 ( Ga5 US :X Ga- S n S nTnT" X7US9n
Sn U
H n
XS S S -
n M Uc W
S nUc W
S nU R 5 nUS :X a/ US!-
n[ R " SU- W
S USS S"SS.S S S#./S$9v OUU S%3-
nT" U5 nSR [ S5 Vs/ s H8 n[ R
" [ R [ R - 5 PM: sn5 n U b S
US U - S
USS.S./0nOSUSS.0n[ R " SU- SUW
S S [ U
S S S 5 S"S S.UES./S9v TT/nT" X7US9n
US :X Ga Sn S&/ Snnn[! U
5 GH u nn
XS S S -
n U( a UR# U
S S S R S'5 5 UR% SR U5 5 ( a Mk UR' 5 [ U5 S : aZ [ R " SU- SUU
S S [ U
S S S 5 S"UR' S 5 S.S./S9v [ U5 S : a MZ SnO=U
S S S S:X a. S nUR# U
S S S R S'5 5 GM$ [ U5 S :X d GM6 [ U
S S S 5 S : d GMQ [ R " SU- SUU
S S [ U
S S S 5 S"US : a U
S S S OU
S S S R) 5 S.S./S9v GM S(U ; d S)U ; a U R+ S*5 ( a U S [ S*5 * R 5 nODU R+ S+5 ( a U S [ S+5 * R 5 nOU R 5 nUU S&3-
nGO[ R " SU- U
S USS 0 S S,S#./S$9v g Sn U
H n
XS S S -
n [ U
S S S R 5 5 S : d M8 U b( S
US W - S
S U
S S S R 5 S.S./0nO SS U
S S S R 5 S.0n[ R " SU- SUU
S S [ U
S S S 5 S S S.UES./S9v M X;R 5 -
nS nT" X7US9n
U SR U
V
s/ s H
oS S S PM sn
5 -
n S(U ; d S)U ; a U b US&-
nUS--
nO1[ R " SU- SUW
S S U b S
OSS S S S S S .S ./S9v g GM* g g s snf s snf s sn
f 7f).Nre z!Streaming for v1 is not supportedrX rn r rO r r, r< Tr r call_r r r r% r r r r r3 r )r r= r r Fr r r= r r r allrp r r r
<|content|>
<|from|>assistant
<|recipient|>rm <|from|> assistant<|from|>assistant
<|from|>assistant
<|from|> assistant
r, r )r" r$ r rangerandomchoicestring
ascii_lettersdigitsrP "CreateChatCompletionStreamResponserstriprF r r~ r r rg
startswithpoplstripendswith) r& r$ r% r chunk_id
chunk_createdr< stopsr r r r] r r func_call_dict
tool_indexr8
stop_sequencebufferis_endr cleaned_completion_text
CONTENT_TOKEN
FROM_TOKENRECIPIENT_TOKEN
STOP_TOKENr0 rr rm s rF generate_streaming:functionary_v1_v2_chat_handler..generate_streaming sV
$C CC", mT * * v./r- A AF!-"78G ,EggNSTViXi v 3 3fmm CDiXG +&gVJ OE# ( (-.*1G*;,60=f0E571& !"
+ * ,(5f(=-/. * &HH!E$K/ 6 %i 0 #Gn *+,0,0/3*" '5*"
! " "E$$)*&-&7(2,016y1A!1DV1L1S1S1U-"
' &N ($().y)9!)v+FF! "MM% *:1*=f*E*K*KC*P Q , 7 7 H H ( &
&)&kAo*5*X*X+1H+(5&+Gn 234]^cdm^nop^qr|^}4~4? 45q5 16i0@0CF0K5:95Ea5H4:6228&( 2*
%&)" # A %:p -?./A*334KLL6E ?3'>#?"?7 #eg 4 - 5 56N O O6E @3'?#@"@7 #eg 4 7F6K6K6M 3 %<$==_"`` * L L % 0"'.$1#: ./-/045; !"%
')O!+'+;A+>v+FF uY / 26 : A A CDqH $ 0$05?29G2C4>8<=B9=Ma=P4:>228&( 9.
)* 3&2"" %4045:95Ea5H,26**0&( 6&2" #."P"P#)H#4'>(5&+Gn 234]^cdm^nop^qr|^}4~487;2* /=2*%&
)" # 9 ",Z 3 3 55F "G!2 %7"J $rwwBLM*y)!,V4*M( O -?./A + "FF"a
* L L % 0#:$1"'. ./ ,1+< )5-<0404379=6: .&!"% . O 1H
+[ YfD NsO A] ?]
C ] D>]>?]=E%]$A
]6]E])B4] ]
1A(])r& r$ r% r r rn rC ry r completion_tokensrm r r{ r| r r r~ r r} rz rx r r r r r% r r r r r= r r )r r= r r r rT rU rV rW )NNr ) rD rK rB r< rm zLiteral['v1', 'v2']r$ rY r& rN r' zUnion[Dict, str]rA )
tokenizer_hasattrrE r< rc additional_special_tokensr" r~ rF r0 r r r$ rg replacer r r r r r r r r r r rP r[ r )?rC rD r$ r% r& r' r( r) r* r5 r6 r+ r, r. r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< rE rB r< END_SYSTEM_TOKENEND_USER_TOKENEND_ASSISTANT_TOKENEND_FUNCTION_RESULT_TOKENSTART_FUNCTION_CALL_TOKENEND_FUNCTION_CALL_TOKENrN r3 r r\ r function_callsfunction_bodiesr r r r r] r8 r r r_ r function_call_dictr r r r rM r0 r> r' r rr rm s? ` ` ` `````` `````````` @@@@@@@@@@@rF functionary_v1_v2_chat_handlerr s/
< zN I > xwx +#y'='='W'WW.* 4$@!$@!"< )
%
,. ,>A , ,B"" L JN@D(.
=
@=
=
%=
G =
> =
&
=
=
~
27V%$4<:;U%T*%%V %k377K[=T
"
+)Wi
F
)"3s9~7J d?&D D ( (F*/*A*A +
+
#+
+
+
+
+
+
+
"+
.+
0+
*+
+
( +
& +
&!+
" #+
$ .%+
& '+
'* U?/ :1=fELLN
,I 6q 9& A ++JSYZZ B 8 G&("ONA A F U!9MRX
d?----62I 23M400 6 7
f8M7NcRR/ -f 5
%%m4%m4 23*&ugVJ ( 3A 6v >O G!45H!I I *7-_D;L
7#$782:fMM *7- @#++45Q79R %%#))*CDRH"MSSU &nR&89 .!(?
"Z%89L%MM! &&z)'? -..]6 2 32m_EE -f 5
%%m4%m4#Z0 .!w
#-Y"7":6"B!Z%89L%MM! &&'<'<'>?M3//MV4K "G )E!2 %E7"J '1&;A&>v&FO%G)<=P)QQ%$3$9$9$;M$- "44(7(=(=(?
]O="AA&--m<"-m"< ,j 9E!2 %E7"J '1&;A&>v&FO%G)<=P)QQ%$-*334KLL #7V#>U:V9V'W WG*334LMM #"c/&::::FH
,/,P (M= !gg &+2Y%. #MM&*>*> *N O%. ' -%2!
-Q2
z?Q 3="<0 'qM* 5f =!+A z!:;!G7"?3 4E
7 /077
4(($ y) W% I*U^J_`aJbcmJn o ++2b=4g -
69_q5H\f W%#
M WPs ?
] ]?] c \ rS rSr% SrS\S' SrS S S jjrS S jrS S jr S!S jr
S
S
S
S
S S S
SSS/ S
S
S
SSSSSSSS
S
S
S
S
S
S. S"S jjr \ S S j5 r
\ S#S j5 r\ S$S j5 r\ S% S&S jj5 rS rg
)'Llava15ChatHandleric
zA chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.rV DEFAULT_SYSTEM_MESSAGEa% {% for message in messages %}{% if message.role == 'system' %}{{ message.content }}{% endif %}{% if message.role == 'user' %}{% if message.content is string %}
USER: {{ message.content }}{% endif %}{% if message.content is iterable %}
USER: {% for content in message.content %}{% if content.type == 'image_url' and content.image_url is string %}{{ content.image_url }}{% endif %}{% if content.type == 'image_url' and content.image_url is mapping %}{{ content.image_url.url }}{% endif %}{% endfor %}{% for content in message.content %}{% if content.type == 'text' %}{{ content.text }}{% endif %}{% endfor %}{% endif %}{% endif %}{% if message.role == 'assistant' and message.content is not none %}
ASSISTANT: {{ message.content }}{% endif %}{% endfor %}{% if add_generation_prompt %}
ASSISTANT: {% endif %}c SS K Jn Xl X l X0l [ 5 U l S U l [ R R U5 ( d [ SU 35 eg )Nr z Clip model path does not exist: ) llama_cpp.mtmd_cppmtmd_cppclip_model_pathr
_mtmd_cppr _exit_stackmtmd_ctxospathexistsrm )rB r r
r s rF r Llava15ChatHandler.__init__
sR - . !$;;?
ww~~o.. ??P QR R /rI c ^ T R b g[ T R S9 T R R 5 nSUl T R Ul UR Ul T R ( a SOSUl T R R T R R 5 UR U5 T l T R c [ ST R 35 eT R R T R 5 ( d [ S5 eU 4S jnT R R! U5 SSS5 g! , ( d f g= f) z-Initialize mtmd context with the llama model.NrQ Trl r z"Failed to load mtmd context from: z%Vision is not supported by this modelc > [ T R S9 T R b, T R R T R 5 S T l S S S 5 g ! , ( d f g = f)NrQ )r r
r r mtmd_freer s rF r 8Llava15ChatHandler._init_mtmd_context..mtmd_free
sC +DLLA}}0 00 ?(,
BAAs :A
A&)r r r
r mtmd_context_params_defaultuse_gpu
print_timings n_threads verbositymtmd_init_from_filer r! r3 rm mtmd_support_visionr callback)rB llama_model
ctx_paramsr s ` rF _init_mtmd_context%Llava15ChatHandler._init_mtmd_context
s
== $
#DLL
9 CCEJ!%J '+||J $#.#8#8J (, 1!J !NN > >$$++-!! DM }}$ #EdFZFZE[!\]] >>55dmmDD !HII
-
% %i 09 :
9
9s D E
Ec $ U R U5 $ r@ ) _load_image)rB image_urls rF
load_image Llava15ChatHandler.load_image
s **rI c r U R c [ S5 e[ U R S9 U R R U R [ R [ U5 - R [ U5 5 [ U5 5 nUc [ S5 eUsSSS5 $ ! , ( d f g= f)z$Create mtmd_bitmap from image bytes.Nz mtmd context not initializedrQ z(Failed to create bitmap from image bytes) r rm r r
r mtmd_helper_bitmap_init_from_bufctypesc_uint8rF from_buffer bytearray)rB image_bytesbitmaps rF _create_bitmap_from_bytes,Llava15ChatHandler._create_bitmap_from_bytes
s == ;< <
#DLL
9^^DD
#k"22?? +@VWK F ~ !KLL :
9
9s A2B((
B6Nr r r r r! Fr r r r" r# r c
^? U R U5 U R c e[ U5 n U S:X a0 U R b# [ R
" SU R S9/U- nU R
U5 n [ SSS9R U R 5 n!U R R 5 R S5 n"U!R USUR " UR " 5 /5 UR " UR " 5 /5 S9n#U H n$U#R# U$U"5 n#M U R$ ( a [' U#[( R* S9 / n%/ n& U HG n$U R- U$5 n'U R/ U'5 n(U%R1 U(5 U&R1 U(5 MI U R R3 5 n)U#R5 S5 U)l SU)l SU)l U R R= 5 n*U*c [? S 5 e U R R@ [C U%5 - " U%6 n+U R RE U R U*[F RH " U)5 U+[C U%5 5 n,U,S
:w a [? S U, 35 eURJ " 5 URL RO 5 [P RR " S
5 n-U R RU U*5 n.[W U.5 GH n/U R RY U*U/5 n0U0c M% U R R[ U05 n1U1U R R\ :X Ga [F R^ " 5 n2U R Ra U0[F RH " U25 5 n3U3( a U2Rb S
: a [W U2Rb 5 V4s/ s H n4U3U4 PM
n5n4URd [C U55 - URf " 5 : a6 [? S URd [C U55 - S
URf " 5 35 eURh " U55 GMU GMX GM[ U1U R Rj U R Rl 4; d GM U R Ro U05 n6URd U6- URf " 5 : a- [? S URd U6- S
URf " 5 35 e[P RR " S
5 n7U R Rq U R URL Rr U0[P RR " URd 5 [P Rt " S
5 URv S[F RH " U75 5 n,U,S
:w a [? SU, 35 eU7Rb Ul2 GM URx S URd R{ 5 n8U R R} U*5 U& H n(U R R U(5 M Ub US S:X a [ U5 nUb U V9s/ s H n9SU9S.PM
nn9UbH [ U[ 5 ( a US:X d US:X a Un[ U[ 5 ( a SU; a
SSUS 0S.nS n:Ub [ U[ 5 ( ax Ubu US S m?[ U?4S jU 5 S 5 n:U:c [? ST? S35 eU:S S n; [ R R [ R " U;5 UR$ S9nUR " S30 S U8_S U_S U_S U _S U
_S!U _S"U ( a U OS _S#U _S$U
_S%U_S&U_S'U_S(U_S)U_S*U_S+U_S,U_S-U_S.U_S/U_S0U_S1U_6n=U:b U:S S n>[ U>U=U 5 $ [ U=U S29$ s sn4f ! U R R} U*5 f = f! U& H n(U R R U(5 M f = fs sn9f ! [ as n Fz%Failed to evaluate chunk: error code r r r r r r rn c 3 B ># U H oS S T:X d M Uv M g7fr rA r s rF r .Llava15ChatHandler.__call__.. r r r r r r r r( r) r* r5 r6 r= r+ r, r- r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< r4 r rA )Or r re r rP rK get_image_urlsr r CHAT_FORMATr mtmd_default_markerdecoder
detokenize token_eos token_bosr r
r, r- r. r r rg mtmd_input_textr! r add_special
parse_specialmtmd_input_chunks_initrm mtmd_bitmap_p_ctypesrF
mtmd_tokenizer byrefreset_ctxkv_cache_clear llama_cpp llama_posmtmd_input_chunks_sizer mtmd_input_chunks_getmtmd_input_chunk_get_typeMTMD_INPUT_CHUNK_TYPE_TEXTc_size_t mtmd_input_chunk_get_tokens_textvaluen_tokensn_ctxevalMTMD_INPUT_CHUNK_TYPE_IMAGEMTMD_INPUT_CHUNK_TYPE_AUDIO mtmd_input_chunk_get_n_tokens mtmd_helper_eval_chunk_singlectx llama_seq_idn_batch input_idstolistmtmd_input_chunks_freemtmd_bitmap_freer# r" r~ r$ r% r& r' r( r) r* r+ r/ r0 r r )@rB rC rD r$ r% r& r' r( r) r* r5 r6 r+ r, r- r. r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< r4 r= r> rE
system_prompt
image_urlsr media_markerr r bitmapsbitmap_cleanupr r
input_textr bitmap_arrayr1 n_pastn_chunksr r
chunk_type n_tokens_out
tokens_ptrjr chunk_n_tokens
new_n_pastr r r3 r4 r5 r r rn s@ @rF rG Llava15ChatHandler.__call__
sC P
&}}(((+H5
B 4#>#>#J>>!4+F+F H ((2
0
+d&&
' ~~99;BB7K "&&&(9':;&&(9':;
$I<< <8D $ << $SZZ ( d 8 ' "ooi 8 77 Dv& %%f- ( 779J "kk' 2JO%)J "'+J $ ^^:: $ C Cc'l RU\ ] 55MMLL, L Q;$'LVH%UVV
))+ #,,Q/ >>@@H x A NN @ @ KE } !% !I!I%!PJ!T^^%N%NN'-'8 %)^^%T%T !6<< #=&
&,*<*>N:U[[]J","8 .9X8YY\]b]h]h]j\k l# &/%8%8%;
!% !M!M MM !JJNN ! % / / ? % 2 21 5 !MM ! "LL 4 " "Q;",/TU[T\-]"^ ^ *4)9)9c )h )95>>:AAC 55f= )//7 ) &?6+Bm+S2?CG !*
!*H ' ( !*
$--- '=F+B + -..6]3J &
f 5! #;--! z*62DKEKTRD| =6K!LMM*%l3F
'44EEJJv&
F % 6 6
#
&.\4
"
.
0
*
(!
" &#
$ &%
& '
( .)
* +
, "-
2 Z(0I7/
++?OO &XN 55f= )//7 )L
==#a&szz2'44@@!++U]] A
s_ B3_ F4^2 ^-B,^2 E!^2 $_ _;4<` -^2 2 __ &_8
a=
A(a88a=c U R S5 ( a) SS KnUR U R S5 S 5 nU$ SS KnUR
R
U 5 nUR 5 nUsS S S 5 $ ! , ( d f g = f)Nzdata:r ,r )r base64 b64decoder urllib.requestrequesturlopenread)r r r urllibr s rF r Llava15ChatHandler._load_image sm ( ( **9??3+?+BCK ! '' 2a ffh " 322s A;;
B c J / nU H nUS S:X d M US c M US H{ n[ U[ 5 ( d M SU; d M" US S:X d M- [ US [ 5 ( a" SUS ; a UR US S 5 Mg UR US 5 M} M U$ )Nr r r r r url)r" r$ rg )rD r r r s rF r !Llava15ChatHandler.get_image_urls s "
Gv&(9%- &y 1G!'400Vw5F"6?k9 *7;+? F F$)W[-A$A * 1 1'+2Fu2M N * 1 1'+2F G 2 rI c
SS jn/ nU nU( ar U" XA5 u pVUbG UbD US: a UR SUSU 45 UR SX 45 XE[ X 5 - S nOUR SU45 SnU( a Mr U$ )z8This method is no longer used in the new implementation.c d [ U5 H! u p#U R U5 nUS:w d M XB4s $ g)Nr rX )r find)ssubstrsr substrposs rF
find_first?Llava15ChatHandler.split_text_on_image_urls..find_first s3 &w / ffVn"9 6M 0 rI Nr r r rO )r$ r~ r% List[str])rg rF )r r r(
split_text remainingr' r s rF split_text_on_image_urls+Llava15ChatHandler.split_text_on_image_urls s FH
6FC1=7 %%vy#&?@!!;
">? %C
,>&>&@ A !!69"56 i rI c
. SS K nSSKJn SSKJn Jn
SSKJn U " U5 U
" 5 n U R U5 V
s/ s H n
[ U
[ 5 ( a U
S OU
PM! nn
/ nU H4 n
U" U
5 R U5 nUR [ U5 5 M6 U V
s/ s H oR X5 ( d M U
PM nn
[ U5 S:X a( [! SU SU S [" R$ " U5 35 e[ U5 S
: a( [! S U S U S [" R$ " U5 35 eUu n[ U" U5 R& 5 nU" U5 R( nU " UUU[+ [, [ US 4 U5 U[+ [, [ US 4 U5 S
9 Uc% U " UUUUU[+ [, [ US 4 U5 SS9nO [. R0 R3 X25 nU " SSU0UD6$ ! [ a [ S5 ef = fs sn
f s sn
f )Nr )Path)hf_hub_download HfFileSystem)validate_repo_idzrLlama.from_pretrained requires the huggingface-hub package. You can install it with `pip install huggingface-hub`.rn zNo file found in z that match z
Available Files:
r zMultiple files found in z
matching )repo_idfilename subfolder local_dirlocal_dir_use_symlinks cache_dirT)r4 r5 r6 r7 r8 r9 local_files_onlyr rA )fnmatchpathlibr0 huggingface_hubr1 r2 huggingface_hub.utilsr3 ImportErrorlsr" r$ relative_torg r~ rF rm r) r* parentrn r r
r r r )clsr4 r5 r7 r8 r9 rE r; r0 r1 r2 r3 hffsr files file_listrel_pathmatching_files
matching_filer6
model_paths rF rF "Llava15ChatHandler.from_pretrained sH E > !~ (
( 'tT 2 2DL <(
" DDz--g6H S] + ,5X948W$9X ~ ! # #G9L
C%%)ZZ %:$;=
~ " *7):hZ H%%)ZZ%6$79
* ]+223
&++ 5dD 19=#95dD 19=
( !##'= uS$_ 5yA!%J i:J
&
I
Ys G4 &H
$HH4H
)r r r r r
T)r r~ r
rR )r rJ )r r~ r[ r )r r r9 )rD rK )r r~ r r* )Nr N)r4 r~ r5 rV r7 &Optional[Union[str, os.PathLike[str]]]r8 z Union[bool, Literal['auto']]r9 rM rE r r[ z'Llava15ChatHandler')r^ r_ r` ra r r r r r r r rG r r r r- classmethodrF rc rA rI rF r r c
s g M
F
S!1F+. IMQU@DLP 02 "
$("%#& # ! ! #@D0415#'&*CEP EP A EP
F EP O
EP >EP JEP EP EP EP EP EP EP . EP !EP"
#EP( ")EP* +EP, !-EP. /EP0 1EP2 3EP4 5EP6 7EP8 9EP: >;EP< .=EP> /?EP@ !AEPB $CEPF
GEPN
#
# $ 0
=A?E<@
Q
Q
Q
: Q
!= Q
:
Q
Q
Q
Q
rI r c \ rS rSrSrSrg)ObsidianChatHandlerid a< {% for message in messages %}{% if message.role == 'system' %}<|im_start|>system
{{ message.content }}
###
{% endif %}{% if message.role == 'user' %}<|im_start|>user
{% if message.content is string %}{{ message.content }}{% endif %}{% if message.content is iterable %}{% for content in message.content %}{% if content.type == 'image_url' and content.image_url is string %}{{ content.image_url }}{% endif %}{% if content.type == 'image_url' and content.image_url is mapping %}{{ content.image_url.url }}{% endif %}{% endfor %}{% for content in message.content %}{% if content.type == 'text' %}{{ content.text }}{% endif %}{% endfor %}{% endif %}###
{% endif %}{% if message.role == 'assistant' %}<|im_start|>assistant
{{ message.content }}###
{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
{% endif %}rA Nr^ r_ r` ra r rc rA rI rF rP rP d s ( rI rP c \ rS rSrSrSrg)MoondreamChatHandleri a {% for message in messages %}{% if message.role == 'user' %}{% if message.content is iterable %}{% for content in message.content %}{% if content.type == 'image_url' %}{% if content.image_url is string %}{{ content.image_url }}
{% endif %}{% if content.image_url is mapping %}{{ content.image_url.url }}
{% endif %}{% endif %}{% endfor %}{% for content in message.content %}{% if content.type == 'text' %}Question: {{ content.text }}
{% endif %}{% endfor %}{% endif %}{% if message.content is string %}Question: {{ message.content }}
{% endif %}{% endif %}{% if message.role == 'assistant' %}Answer:{{ message.content }}
{% endif %}{% endfor %}{% if add_generation_prompt %}Answer:{% endif %}rA NrQ rA rI rF rS rS s " rI rS c \ rS rSrSrSrSrg)Llava16ChatHandleri zA chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. a {% for message in messages %}{% if message.role == 'system' %}{{ message.content }}{% endif %}{% if message.role == 'user' %}{% if message.content is iterable %}{% for content in message.content %}{% if content.type == 'image_url' %}{% if content.image_url is string %}{{ content.image_url }}
{% endif %}{% if content.image_url is mapping %}{{ content.image_url.url }}
{% endif %}{% endif %}{% endfor %}{% for content in message.content %}{% if content.type == 'text' %}{{ content.text }}{% endif %}{% endfor %}{% endif %}{% if message.content is string %}{{ message.content }}{% endif %}{% endif %}{% if message.role == 'assistant' %}{{ message.content }}{% endif %}{% endfor %}{% if add_generation_prompt %}Answer:{% endif %}rA Nr^ r_ r` ra r r rc rA rI rF rU rU s } % rI rU c \ rS rSrSrSrSrg)NanoLlavaChatHandleri zAnswer the questionaM {% for message in messages %}{% if message.role == 'system' %}<|im_start|>system
{{ message.content }}<|im_end|>{% endif %}{% if message.role == 'user' %}<|im_start|>user
{% if message.content is string %}{{ message.content }}{% endif %}{% if message.content is iterable %}{% for content in message.content %}{% if content.type == 'image_url' and content.image_url is string %}{{ content.image_url }}{% endif %}{% if content.type == 'image_url' and content.image_url is mapping %}{{ content.image_url.url }}{% endif %}{% endfor %}{% for content in message.content %}{% if content.type == 'text' %}{{ content.text }}{% endif %}{% endfor %}{% endif %}<|im_end|>{% endif %}{% if message.role == 'assistant' %}<|im_start|>assistant
{{ message.content }}<|im_end|>{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
{% endif %}rA NrV rA rI rF rX rX s 3( rI rX c \ rS rSrSrSrSrg) Llama3VisionAlphaChatHandleri-
Na {% for message in messages %}<|start_header_id|>{% if message.role == 'user' %}user<|end_header_id|>
{% if message.content is iterable %}{% for content in message.content %}{% if content.type == 'image_url' %}{% if content.image_url is string %}{{ content.image_url }}{% endif %}{% if content.image_url is mapping %}{{ content.image_url.url }}{% endif %}{% endif %}{% endfor %}{% for content in message.content %}{% if content.type == 'text' %}{{ content.text }}{% endif %}{% endfor %}{% endif %}{% if message.content is string %}{{ message.content }}{% endif %}{% endif %}{% if message.role == 'assistant' %}assistant<|end_header_id|>
{{ message.content }}{% endif %}<|eot_id|>{% endfor %}{% if add_generation_prompt %}<|start_header_id|>assistant<|end_header_id|>
{% endif %}rA rV rA rI rF rZ rZ -
s "& rI rZ c \ rS rSrSrSrSrg)MiniCPMv26ChatHandlerib
r a {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}<|im_start|>system
You are a helpful assistant.<|im_end|>
{% endif %}<|im_start|>{{ message['role'] }}
{% if message['content'] is iterable %}{% for content in message['content'] %}{% if content.type == 'image_url' %}{% if content.image_url is string %}{{ content.image_url }}{% endif %}{% if content.image_url is mapping %}{{ content.image_url.url }}{% endif %}{% endif %}{% endfor %}{% for content in message['content'] %}{% if content.type == 'text' %}{{ content.text }}{% endif %}{% endfor %}{% endif %}{% if message['content'] is string %}{{ message['content'] }}{% endif %}<|im_end|>
{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
{% endif %}rA NrV rA rI rF r\ r\ b
s ; rI r\ c 0 ^ \ rS rSrSrSrU 4S jrSrU =r$ )Qwen25VLChatHandleri
r aO {% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
{{ self.DEFAULT_SYSTEM_MESSAGE }}<|im_end|>
{% endif %}<|im_start|>{{ message['role'] }}
{% if message['content'] is string %}{{ message['content'] }}<|im_end|>
{% else %}{% for content in message['content'] %}{% if content['type'] == 'image_url' %}{% if content.image_url is string %}{{ content.image_url }}{% else %}{{ content.image_url.url }}{% endif %}{% elif content['type'] == 'text' %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
{% endif %}{% endfor %}<|im_start|>assistant
c > US nUR " 5 UR R 5 SUl [ US5 ( a UR
R
S5 [ U S5 ( a S U l S U l U R ( aH UR S/ 5 n[ U R U5 5 n[ SU S3[ R S9 [ TU ]D " S 0 UD6$ )
NrC r r _last_image_embedrD z$Minimal - Cleared state, processing z imagesr rA )r r r r r r fillr` _last_image_hashr
r
rF r r, r- r. superrG )rB rE rC rD image_count __class__s rF rG Qwen25VLChatHandler.__call__
s w
!!# 5+ & & OO # 4, - -%)D "$(D ! << zz*b1H d 1 1( ;U&S? - S@U&SA U&S SBS [= U&S5 S S# 5 SCS [? [A U)U(5 5 V,V-V&s/ s H. u n,u n-n&SDSEU, SE3- U-- SE- U&S? - SU-U&S5 S S6 S=.SF.PM0 sn&n-n,SG.U+ESH./[C SI U( 5 5 [C SJ U( 5 5 [C SK U( 5 5 SL.SM.$ [ SN5 es sn f ! [* ah n![ R R [ R, U R( S+9nU R( ( a [/ S,5 [/ U!5 S n!A!GNS n!A!ff = fs sn f ! [* ah n![ R R [ R, U R( S+9nU R( ( a [/ S,5 [/ U!5 S n!A!GN'S n!A!ff = fs sn&n-n,f )PNa {% for message in messages %}<|im_start|>{{ message.role }}
{% if message.role == 'system' %}{{ message.content }}{% if tool_calls %}
You have access to the following functions:
{% for tool in tools %}
functions.{{ tool.function.name }}:
{{ tool.function.parameters | tojson }}
{% endfor %}
You can respond to users messages with either a single message or one or more function calls.
To respond with a message begin the message with 'message:', use the following format:
message:
To respond with one or more function calls begin the message with 'functions.:', use the following format:
functions.:
{ "arg1": "value1", "arg2": "value2" }
functions.:
{ "arg1": "value1", "arg2": "value2" }{% endif %}<|im_end|>
{% endif %}{% if message.role == 'user' %}{{ message.content }}<|im_end|>
{% endif %}{% if message.role == 'assistant' %}{% if message.content and message.content | length > 0 %}{% if tool_calls %}message:
{% endif %}{{ message.content }}<|im_end|>
{% endif %}{% if 'tool_calls' in message %}{% for tool_call in message.tool_calls %}functions.{{ tool_call.function.name }}:
{{ tool_call.function.arguments }}{% endfor %}<|im_end|>
{% endif %}{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
{% endif %}htmlxml)
autoescape undefinedr r r r rn r r T)rD r& r r r r r r( r) r* r5 r6 r+ r, r/ r0 r1 r2 r7 r8 r9 r: r3 r; r< r= r c 3 B ># U H oS S T:X d M Uv M g7fr rA r r3 r s rF r *chatml_function_calling..i s" MedJ'7'?9'LTTer zTool with name 'z' not found in toolsrA rC r r rS r z "functions.z:"z0root ::= functions | "message:"
functions ::= r z2root ::= functions | "<|im_end|>"
functions ::= Frx r r r z message:
c 3 B ># U H oS S T:X d M Uv M g7fr rA rn s rF r ro " Q%$
+;F+Cy+P%r c 3 B ># U H oS S T:X d M Uv M g7fr rA rn s rF r ro 0 rq r r r% r r r r r r r rz r r )r r r )r r r= r c 3 B # U H nS U; a US S OSv M g7f)r r r NrA r r s rF r ro g s8 ) '2
#j0 #7+,?@ '2 c 3 B # U H nS U; a US S OSv M g7f)r
prompt_tokensr NrA rt s rF r ro o s/ %&1
=Dz rE function_calling_templatetemplate_rendererr r r3 r5 r function_namesinitial_gbnf_tool_grammarfollow_up_gbnf_tool_grammarr r completionscompletions_tool_nameresponser r r s. `rF chatml_function_callingr
s F0 K f 6++VUO<(( k+, &
& # $
&
mS ) ) V #}'>'K mT * *v /F"M& 1 K dC
| &*T\N
" {C ( ([F-B = u:?")) "& *
&?6+Bm+S2?CG* # #
'
$
&
"2
#4
.
,
*
*!
" #
$ "2%
& '
( *2 t)
, /
6 +t$$
+F3 Me Mt
< / {:N OP P")) "& *
Jyk-- #00AA
4
+L9:EMM B G % 6 6
#
"
.
0
*
(
&
&!
" #
$ .%
& '
* 4 +V
k3 ' 'K6,AA AZZDIJED[j)&12" 7EJN )*" 0
)*" 0
%
% " & F !22
U * , & $ " "! " # $ *% & **66 %u}} 7
' . 8LJ i #F +DD* # #
,
'
$
#^
*2 t
"2
#4
.
,
*!
" *#
$ %
& "2'
( &22>>/ ? )
0 3
: S &()I Q%QSW XD BD +-
9+S1 1F
'44EEJJtJ / =>
F $)#:#: $ $ '$ $ $
$ $
$ $ $ $ "2$ #4$ .$ $ , $ * $ *!$" #$$ "2%$& '$ * $(446J$
3 4 ! ( ( 3 *95a8@ @F dNF .. '
$
"2 #4 . , * *!" #$ "2%& &22>>/ ? 'H. K @ @(KH +A.v6s<7H7JKIQ%QSWDM p ; 1$ %!,Q !:1!=f!E"
" :d++'!), ( &2 I*U^J_`aJbcmJn o +#' ?H #$9; G? '? :#:Iz '.$%aS('+"+', #&'& #-T"2 '3
)3,51;I1Fq1I&1Q-" ? ' ( -) < &) ) '2
) & "% %&1% " !$ $&1$ !E4
4
l G
HHy
t #00<< '' = G }}c a J Kd
'44@@!++U]] A ==g !H
D'sF Z($AZ- (\"+A\' %5^ -
\ 7A \\ '
^1A ^^r r )r= z(Optional[llama_types.CompletionLogprobs]r[ z"llama_types.ChatCompletionLogprobs)r zllama_types.Completionr[ zllama_types.ChatCompletion)r r r[ z)Iterator[llama_types.ChatCompletionChunk]r} )r aUnion[llama_types.CreateCompletionResponse, Iterator[llama_types.CreateCompletionStreamResponse]]r+ rR r[ zZUnion[llama_types.CreateChatCompletionResponse, Iterator[llama_types.ChatCompletionChunk]])r r~ r r r+ rR )r6 r r[ r )rG Union[str, os.PathLike[str]]r[ r )rG r r[ r rL )rU Dict[str, Any]r rR r[ r )rU r r rR r[ r )ra Dict[str, str]r[ rV )rD rK r[ r~ )rD rK rh r r[ List[Tuple[str, Optional[str]]])
ro r~ rD r rp r~ rq r~ r[ r~ )ro r~ rD r rp r~ r[ r~ )r
rR )FT)r4 r~ r
rR r rR )r. z/llama_types.ChatCompletionRequestResponseFormatr
rR r )rD z.list[llama_types.ChatCompletionRequestMessage]rE r r[ r )4rC rJ rD rK r$ rL r% rM r& rN r' rO r( rP r) rP r* rQ r5 rP r6 rP r+ rR r, rS r. rU r/ rT r0 rP r1 rP r2 rP r7 rP r8 rQ r9 rP r: rP r3 rV r; rX r< rY r[ zLUnion[llama_types.ChatCompletion, Iterator[llama_types.ChatCompletionChunk]])8rC rJ rD rK r$ rL r% rM r& rN r' rO r( rP r) rP r* rQ r5 rP r6 rP r+ rR r, rS r. rU r/ rT r0 rP r1 rP r2 rP r7 rP r8 rQ r9 rP r: rP r3 rV r; rX r< rY r= rZ r> rT r[ r\ )x
__future__r r r- r) r dataclassesr r r
contextlibr typingr r r r r
r r r
r r r jinja2.sandboxr numpynp numpy.typingnptllama_cpp.llama_cppr llama_cpp.llamarC llama_cpp.llama_typesrP llama_cpp.llama_grammarr& _loggerr _utilsr r r] CHATML_BOS_TOKENCHATML_EOS_TOKENr^ MISTRAL_INSTRUCT_BOS_TOKENMISTRAL_INSTRUCT_EOS_TOKENr_ r` r r+ re rh r rq dataclassr r r r r r r r r rH rJ rV rX rb re rj ru rz r| r~ r r r r r r# r r r r r r r r r r r r r r r r r r r r r r r r ra r r rP rS rU rX rZ Llama3VisionAlphar\ r^ r rA rI rF r s
"
8 ' + / 5
a "A " # "z !l
0 0 f ) B
$H $ H?- H?V6'2& 0(
@(
.(
` <<
<
<"n;n;n;
n;bR#!R# R#j #? 2E#?E E #'-#$-# -# -#d #'E$E E E.<
<
%
#BILTW"
#B
IL
#B IL TW
#B
IL
#B
IL
#B
IL
#B IL BF :> 2D 2
2$ i 1< 1 1 1 ! 1 i
<<
<
<
< !
< h
1<
1
1
1
1 f
=<
=
=
=
= h 1< 1 1 1 1 m$ 1< 1 1 1 % 1 l# 1< 1 1 1 $ 1 j! 1< 1 1 1 " 1 k"1<11 1 #1* () =< = = = * = h =<== = =* g
1<
1
1
1
1 g
1<
1
1
1
1 k"@<@@ @ #@4 m$ 1< 1 1 1 % 1 h
<<
<
<
<
< k" << < < < # < h
<<
<
<
<
< ();<;; ; *;( j!
<<
<
<
< "
< j!<<<< < "<& g 9<99 9 9* g <<<< < <( "-0 EIMQ<@HL ,.QU $ ! " <@,03b b<b Bb K b
: b F
b b b b b b
b *b O b b !b" #b$ %b& 'b( )b* +b, -b. /b0 :1b2 *3b6 R7b 1bJ ""23!"23 EIMQ<@HL ,.QU $ ! " <@,03
<
B
K
:
F
*
O
!
" #
$ %
& '
( )
* +
, -
. /
0 :1
2 *3
6 R7
4 4
D ~
~
B5, 5p'- 'T-+ -`4- 4n.#5 .d 1 # . #L7*, 7*t "";< EIMQ<@HL ,.QU $ ! " <@,0 #"&7wI wI<wI BwI K wI
: wI F
wI wI wI wI wI wI
wI *wI O wI wI !wI" #wI$ %wI& 'wI( )wI* +wI, -wI. /wI0 :1wI2 *3wI4 5wI6 7wI:;wI =wIrI