Support OpenAI and Anthropic message processors - #9809
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Thanks @ezeli and @HGY19854338008 for raising #8327 and #9084 and providing concrete OpenAI tool-call examples — they were very helpful when shaping this change. This PR adds normalization for OpenAI @Jintao-Huang @hjh0119 @tastelikefeet, when convenient, we would also appreciate your guidance on whether |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Thanks! I will check this code asap |
| new_messages.append({ | ||
| 'role': 'tool_response', | ||
| 'content': cls._anthropic_block_content(block.get('content', '')), | ||
| **message_metadata, |
There was a problem hiding this comment.
What about the multi-modal blocks?
There was a problem hiding this comment.
Thank you for catching this. The previous implementation only handled text and tool blocks, so Anthropic image blocks could be silently dropped.
I addressed this in 526c412:
- base64 and URL image sources are converted to ms-swift's canonical
<image>placeholders plus the top-levelimagesfield, preserving their original order; - the same conversion works for both regular message content and multimodal
tool_resultcontent; - unsupported image source/content block types now raise a clear error instead of being silently discarded;
- auto-detection also recognizes Anthropic image blocks without requiring a tool block.
The regression test covers a mixed text -> base64 image -> tool_use -> URL image tool_result -> text conversation, verifies the canonical messages and ordered images values, checks StdTemplateInputs compatibility, and loads the base64 image through ms-swift's existing load_image path. Thank you for the helpful review!
There was a problem hiding this comment.
As a follow-up, I also added symmetric regression coverage for OpenAI Chat Completions multimodal messages in 54b24c99. No additional OpenAI processor conversion was needed because the existing StdTemplateInputs.remove_messages_media path already extracts image_url blocks into <image> placeholders and the top-level images field. The new test verifies base64 and URL images in order, real base64 decoding, and that assistant text can coexist with normalized tool_calls. The focused 9-test suite and all pre-commit hooks pass. Responses API input_text/input_image blocks use a separate schema and are documented as outside this Chat Completions example. Thank you again for prompting the multimodal coverage!
|
|
||
| OpenAI tool-call format: | ||
| ```jsonl | ||
| {"messages": [{"role": "user", "content": "How is the weather?"}, {"role": "assistant", "content": null, "tool_calls": [{"type": "function", "function": {"name": "get_weather", "arguments": "{\"city\":\"Beijing\"}"}}]}, {"role": "tool", "content": "Sunny"}]} |
There was a problem hiding this comment.
An example of multi-modal messages is needed
There was a problem hiding this comment.
Thanks — I added matching Anthropic multimodal tool-use examples to both the English and Chinese custom-dataset documentation in 526c412.
The example includes a base64 image in a normal user message and a URL image inside tool_result, and explains that they are converted to <image> placeholders and the top-level images field in their original order.
There was a problem hiding this comment.
I have now added matching OpenAI Chat Completions multimodal examples to both the English and Chinese documentation in 54b24c99. The example covers base64 and URL image_url blocks together with an assistant tool_calls entry, and documents their conversion to ordered <image> placeholders plus the top-level images field. It also clarifies that Responses API input_text/input_image blocks use a different schema. Thanks again for the helpful suggestion!
3 participants
What changed
assistant.tool_callsinto SWIFTtool_callmessagestool_useandtool_resultcontent blocks into SWIFT canonical rolesOpenAIMessagesPreprocessorandAnthropicMessagesPreprocessorclassesMessagesPreprocessorWhy
MessagesPreprocessor._check_messageskeeps onlyrole,content,loss, andloss_scale. OpenAI-style tool calls stored inassistant.tool_callswere therefore silently removed during dataset preprocessing even though templates can encode tool calls. This can produce training examples that retain assistant narration but lose the actual tool-call targets.Anthropic tool-use traces similarly need their content blocks expanded into SWIFT's canonical
assistant,tool_call, andtool_responseroles before template encoding.Impact
OpenAI and Anthropic agent traces can now be passed through the normal dataset preprocessing path without an external conversion script. Parallel OpenAI tool calls, string or object arguments, Anthropic text/tool interleaving, nested text tool results, and loss metadata are covered.
The regression test also decodes supervised labels and asserts that the tool name is present, rather than only checking that template encoding succeeds.
Validation
pre-commit run --all-filespython -m unittest tests.general.test_data_preprocess.TestProviderMessagesPreprocess tests.general.test_data_preprocess.TestRejectedMessagesPreprocess tests.general.test_data_preprocess.TestDataPreprocess.test_tool_messageRelated to #8327 and #9084.