Paper deep dive

UniData: Universal Multimodal Instruction Generation Pipeline

Jiaqi Tang, Yi-Feng Wu, Yuting Zhang, Hao Lu, Bowen Fu, Qing-Guo Chen, Xiaogang Xu, Yuwei Hu, Shiyin Lu, Wei Wei, Lei Zhang, Zhao Xu, Weihua Luo, Qifeng Chen, Ying-Cong Chen

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 96

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 10/9/2026, 8:45:52 AM

Summary

The paper introduces UniData, a universal multimodal instruction generation pipeline that transforms simple user requirements into multi-round, multimodal instructions. It addresses limitations in existing methods regarding modality support and multi-round generation by expanding user inputs into diverse events, using an any-to-any large model (LLaMA-3 backbone) for generation, and correcting inference flows. The authors also present UniDataset, a 20,000-entry dataset across nine modalities (language, image, music, emoji, code, map, link, math, QR code) used to train the pipeline. Experiments show UniData achieves state-of-the-art performance in data quality and enhances the capabilities of other multimodal models.

Entities (14)

UniData · method · 98%UniDataset · dataset · 97%GPT-4o · model · 95%Llama 3 · model · 95%Multimodal Large Language Models · concept · 95%Diverse Event Generator · component · 92%Multimodal Instruction Generator · component · 92%DALL·E 3 · model · 90%MusicGen · model · 90%NExT-GPT · model · 90%Ovis-1.5 · model · 90%Self-Instruct · method · 90%VIGC · method · 90%Whisper · model · 90%

Relation Signals (12)

UniDataset → contains → 20,000 entries

confidence 95% · UniDataset, a dataset comprising 20,000 entries across nine modalities

UniData → generates → multi-round multimodal instructions

confidence 95% · UniData... to transform simple user requirements into multi-round, multimodal instructions.

UniData → supportsmodalities → nine modalities

confidence 95% · UniDataset... comprising 20,000 entries across nine modalities... include<language>,<image>,<music>,<emoji>, <code>,<map>,<link>,<math>, and<QR code>

UniData → uses → UniDataset

confidence 95% · To train this pipeline, we also build UniDataset... Our experiments demonstrate that UniData achieves SOTA performance

UniData → usesbackbone → Llama 3

confidence 95% · we select LLaMA-3 (Grattafiori et al., 2024) as the main backbone of our pipeline.

UniData → usescomponent → Diverse Event Generator

confidence 92% · we build the Diverse Event Generator (DEG) to expand diverse events

UniData → usescomponent → Multimodal Instruction Generator

confidence 92% · we construct a large model for multimodal understanding and generation... Multimodal Instruction Generator (MIG)

UniData → outperforms → VIGC

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs remains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round instructions. To address these problems, we introduce UniData, a universal instruction generation pipeline, to transform simple user requirements into multi-round, multimodal instructions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruction generation. Finally, UniData enhances data quality by correcting irrelevant and redundant inference flow, leveraging correlations between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising 20,000 entries across nine modalities for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation capabilities of other multimodal models.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

92,284 characters extracted from source content.

Expand or collapse full text

UniData: Universal Multimodal Instruction Generation Pipeline Jiaqi Tang 1 * , Yi-Feng Wu 2 * , Yuting Zhang 3 * , Hao Lu 3 , Bowen Fu 4 , Qing-Guo Chen 2 , Xiaogang Xu 5 , Yuwei Hu 2 , Shiyin Lu 2 , Wei Wei 4 , Lei Zhang 4 , Zhao Xu 2 , Weihua Luo 2 , Qifeng Chen 1† , Ying-Cong Chen 3† 1 The Hong Kong University of Science and Technology 2 ATH, Alibaba Group 3 The Hong Kong University of Science and Technology (Guangzhou) 4 Northwestern Polytechnical University 5 Zhejiang University jtang092, cqf@ust.hk, yingcongchen@hkust-gz.edu.cn Abstract Multimodal Large Language Models (MLLMs) are increasingly being applied in a wider range of real-world scenarios. However, due to the substantial labor cost, creating high-quality multimodal instruction datasets for MLLMs re- mains a significant challenge. Although some methods propose to generate instruction data, they often face limitations in modality support and struggle with generating multi-round in- structions. To address these problems, we in- troduce UniData, a universal instruction gen- eration pipeline, to transform simple user re- quirements into multi-round, multimodal in- structions. Specifically, UniData first expands user requirements into multiple diverse events. Using these events, UniData then integrates an any-to-any large model for multimodal instruc- tion generation. Finally, UniData enhances data quality by correcting irrelevant and redun- dant flows during inference, leveraging correla- tions between instruction rounds. To train this pipeline, we also build UniDataset, a dataset comprising20, 000entries across nine modali- ties for improved multimodal generation. Our experiments demonstrate that UniData achieves SOTA performance in data quality and can also enhance the understanding and generation ca- pabilities of other multimodal models. 1 Introduction “In God we trust; all others must bring data.” — W. Edwards Deming Recently, Multimodal Large Language Models (MLLMs) have been extensively developed across various fields (Cui et al., 2024; Li et al., 2024a). * Equal contribution. † Corresponding authors. However, to propel MLLMs toward fulfilling real- world requirements, data scarcity has gradually be- come a significant issue (Liu et al., 2023; Müller and Unay, 2017; Yin et al., 2023; Wang et al., 2024a, 2023). To acquire high-quality data, most previous methods have relied on manual collection and an- notation (Li et al., 2017), which consequently de- mands substantial human resources (Tang et al., 2024; Liu et al., 2024b; Fang et al., 2024). While some approaches (Li et al., 2023b; Liu et al., 2023) have started utilizing GPT (OpenAI, 2024) for au- tomated or semi-automated data generation to alle- viate labor costs, most of them are designed for one specific scenario and cannot generalize to broader applications. In this paper, we propose UniData, a universal multimodal instruction generation pipeline. As shown in Figure 1, by simply inputting basic re- quirements, users can finally obtain large-scale mul- timodal instruction datasets. While some solutions have attempted to gener- ate instructions in a universal setting, such as Self- Instruct (Wang et al., 2023) and VIGC (Wang et al., 2024a), these methods are primarily restricted to language modality (as shown in Figure 2 (A)(B)). Multimodal Self-Instruct (Zhang et al., 2024) at- tempts to generate instructions with corresponding abstract images, but its capabilities are limited to this (as shown in Figure 2 (C)). Furthermore, all of these methods provide limited support for multi- round instruction generation. To address the aforementioned problems, we first focus on enhancing data diversity by expand- ing user keyword requirements into a series of po- tential instructional events, as described in Sec- tion 4.1. Next, to improve multimodal support, we propose an any-to-any large model that assists users arXiv:2610.11363v1 [cs.AI] 8 Oct 2026 (I)Diverse Events RequirementUser An artist and an e-commerce expert discuss how to sell artwork online. They talk about setting ... e-commerce art media EmotionOccasion Online platforms can help artists reach global audiences, explore new creative collaborations ... More ..... · (More) Q: Once I choose a platform, what should I focus on next? A: High-quality images of your artwork are crucial. They need to be well-lit and showcase the details to attract potential buyers.<image> Q: How do I ensure my images meet these standards? A: Invest in a good camera or hire a professional photographer. Editing tools can also help enhance the images, but the initial quality is key. <link> ------------------------------------------------------------------------------------------------ Search: Best cameras for photographing artwork Result: 1.Investing in a camera to photograph my artwork 2.Digital Camera Comparison for Artists | Master Oil Painting 3.What's The Best Photography Camera For Artwork Right Now? ------------------------------------------------------------------------------------------------ (More) ·<emoji> A: Consistency is important. Aim for at least a few posts per week, and make sure to interact with your followers by responding to comments and messages. <music> · (More) (I)Multi-Modal Instructions More ..... (I)User Input with Random Seeds Tune for social media engagement Figure 1: Usage of UniData. (I) Users only input one simple requirement, combined with random emotions and occasions to create diverse events. (I) Each diverse event will generate one multimodal instruction. (I) The generated multimodal instructions include various modalities. in generating multi-round, multimodal instructions iteratively, as described in Section 4.2. Finally, to ensure data quality in a multi-round context, we correct the instruction flow by assessing the cor- relation between instructions from previous and current rounds. This process effectively minimizes redundancy and irrelevance during inference (Sec- tion 4.3). To address the limitations in quantity, quality, and copyright issues of existing data (in Section 3), we also construct UniDataset to support the training of UniData. This dataset comprises 20,000 entries and incorporates nine distinct modalities on both the input and output sides. These modalities in- clude<language>,<image>,<music>,<emoji>, <code>,<map>,<link>,<math>, and<QR code> (see Section 3.1). The inclusion of these diverse modalities is intended to enhance our data genera- tion pipeline, enabling the creation of more com- prehensive multimodal instructions and thereby im- proving overall versatility. Experimental results demonstrate that the quality of our generated data surpasses that of other gener- ation pipelines on all four dimensions we evaluate: Emoji Emoji (A)Language-to-Language (e.g., Self-Instruct) Language Large Language Model Language (B)Multimodal-to-Language (e.g., VIGC) (C) Language-to-Multimodal (e.g., Multimodal Self-Instruct) Large Vision-Language Model Language LanguageImage Language Large Language Model Language Simulated To o l s Abstract Image (D) Multimodal-to-Multimodal (Ours) Language Multi-Modal Large Language Model ImageMusic Language ImageMusic Link Link · · Image Detokenizer Music Detokenizer Emoji Retrieving · Figure 2: Comparison of different instruction genera- tion frameworks. (A) Self-Instruct (Wang et al., 2023) can only support language modality. (B) VIGC (Wang et al., 2024a) integrates visual modality into its genera- tion. (C) Multimodal Self-Instruct (Zhang et al., 2024) can output abstract images. (D) Ours (UniData) can freely understand and generate different modalities in instructions. reasonableness, clarity, detail, and relevance. Fur- thermore, our generated data can also enhance the understanding and generation capabilities of other multimodal large language models. Our contribu- tion is shown as follows: • We propose UniData, a universal multimodal instruction generation pipeline that can create multi-round multimodal instructions from one simple requirement. • We propose UniDataset, a multimodal instruc- tion dataset comprising20, 000instructions with nine distinct modalities in both input and out- put. UniDataset addresses existing limitations in data quantity, quality, and copyright status, and provides a robust data foundation for training UniData or other any-to-any models. • Through comprehensive experiments, we demon- strate that UniData surpasses existing pipelines in data quality, achieving SOTA performance in reasonableness, clarity, detail, and relevance. Ad- ditionally, the generated data can improve the un- derstanding and generation capabilities of other multimodal models. 2 Related Work Multimodal Large Language Models Multi- modal Large Language Models can understand multimodal information (Liang et al., 2024) by integrating various modalities, including but not limited to text, images, audio, and video. Due to their versatility, they have been widely applied in diverse fields, such as embodied intelligence (Li et al., 2024a), video understanding (Tang et al., 2024), and autonomous driving (Cui et al., 2024). Most previous multimodal models (Liu et al., 2023; Li et al., 2023b; Lu et al., 2024b) were de- signed to understand multimodal information, yet their output is limited to language only. Recently, SEED-X (Ge et al., 2024) and Show-o (Xie et al., 2025) have begun to explore the integrated capabil- ities of image understanding and generation. Addi- tionally, NExT-GPT (Wu et al., 2024) has advanced by integrating more modalities, including images, audio, and video, for unified understanding and generation. Nonetheless, training these models encounters substantial challenges, particularly regarding data availability. For example, NExT-GPT (Wu et al., 2024) is unable to provide training data due to copy- right constraints. Furthermore, Show-o (Xie et al., 2025) is limited to training in an end-to-end way due to the lack of an appropriate dataset. Motivated by these challenges, we propose UniData, which is designed to generate multimodal instructions to facilitate the model training process and potentially support further applications. Instruction Generation The scarcity of train- ing data impacts the performance of Multimodal Large Language Models, leading to increased in- terest in how to generate high-quality instruction data. Self-Instruct (Wang et al., 2023) pioneered the generation of diverse instructions using a small set of seed tasks, but it focuses only on the lan- guage modality (Figure 2 (A)). To overcome these limitations, VIGC (Wang et al., 2024a) integrates visual understanding into the instruction genera- tion process (Figure 2 (B)). Besides, Multimodal Self-Instruct (Zhang et al., 2024) addresses the problem of abstract image synthesis in instruction generation, utilizing synthetic tools to create both diagrams and corresponding text (Figure 2 (C)). However, these methods still support limited multi- modal capabilities. To address these constraints, we propose UniData, a universal multimodal instruction gen- eration pipeline designed to both understand and generate nine types of multimodal information (Fig- ure 2 (D)). This approach broadens the applicability of multimodal instruction generation compared to previous research. 3 Dataset Construction To train UniData, a dataset containing both mul- timodal inputs and outputs is essential. However, datasets meeting these criteria are scarce. As sum- marised in Table 10 (Appendix A.3), the majority of existing datasets are confined to a single modal- ity, especially for the output. While NExT-GPT (MosIT) (Wu et al., 2024) and GenHowTo (Sou ˇ cek et al., 2024) have developed some initial datasets encompassing multimodal inputs and outputs, they are hindered by a restricted range of modalities, limited instruction rounds, and a small data size. Furthermore, copyright concerns limit their avail- ability. To enrich our data resources, we have also de- veloped a multimodal dataset named UniDataset (the Ours row in Table 10). This dataset supports inputs and outputs across nine modalities, thereby facilitating the training process of UniData. 3.1 Procedure of Data Construction Stage (I): Preparing Diverse EventsTo accom- modate data diversity, it is crucial to generate var- ied<events>(tasks) for each multi-round instruc- tion. Drawing inspiration from Self-Instruct (Wang et al., 2023), we first create a Topic Pool with90 <topics>spanning multiple domains, such as e- commerce, lifestyle, and sports. Then, we ran- domly pair two topics and input them into GPT- 4o (OpenAI, 2024) to generate diverse<events>, as shown in Figure 3 (I). Compared to a single <topic>, this approach introduces greater random- ness ( 90 2 = 4,005 unique combinations), thereby facilitating data diversity. Stage (I): Generating Multi-Round Instruc- tions After obtaining<events>(tasks), we uti- lize them to generate multi-round instructions. Con- sidering other attributes of instructions, we also ran- domly choose different<rounds>(ranging from5 to30),<emotion>(random proportions of seven distinct emotions), and<occasion>(formal or in- formal) to collectively guide GPT-4o (OpenAI, 2024) in generating diverse instructions, as shown in Figure 3 (I). Ultimately, we generate20, 000 multi-round instructions. Stage (I): Forecasting Multimodal Integration Our current multi-round instructions are restricted to a single modality (<language>). To generate multimodal content, we utilize GPT-4o (OpenAI, 2024) to predict potential multimodal content and Q: Hey everyone, guess what? I just finished my school project on building a robot! A: That's amazing! Tell us more about it. Q: So, it's a small robot that can navigate through a maze using sensors. Isn't that cool? A: Wow, that's really impressive! How did you come up with the idea? Q:Our teacher gave us a list of project ideas· A:· (I)GeneratingMulti-RoundInstructions To p i c TopicsPool Q 1 ∪Q 2 Event A family dinner where a teenager excitedly shares their school project on building a robot, ...... A forensic scientist and a business consultant discuss the integration of forensic science, ...... More...... (I)Preparing Diverse Events RoundEmotion Occasion N = Random(5-30) Formal Conversation OR Informal Conversation EmotionsRate Angry1.37% Disgusted36.82% Fearful17.43% Happy3.92% Neutral16.15% Sad11.45% Surprised12.86% Overall100% Help me generate one instruction event about 퐐 ퟏ and퐐 ퟐ . Please only output the summary of event, without any other extra words, and as simple as possible. OpenAI GPT 4o Please generate a information-rich Instruction with Round rounds. The current event is: Event. The occasion is: Occasion. The emotions are: Emotion. OpenAI GPT 4o Multi-Round Instruction More...... ...... Multi-ModalIndicatorTo k e n s Round1:QA Round2:QA Round3:QA · RoundN:QA ModalityTokenX (Prompt/Query/Others) image<|image|>X<|image_end|>Detail Prompt for Dell-E music<|music|>X<|music_end|> DetailPrompt for MusicGen emoji<|emoji|>X<|emoji_end|>Emoji Label code<|code|>X<|code_end|> PromptforGenerating Code by GPT-4o map<|map|>X<|map_end|>Queryfor GoogleMap link<|link|>X<|link_end|>Query for GoogleSearch math<|math|>X<|math_end|> Prompt for Generating Math Formular by GPT-4o QRcode<|QRcode|>X<|QRcode_end|>Content in QR Code Please integrate multimodal information into the instruction. We currently have EIGHT types of specialtokens, expressed as Tokens . Replace the middle X in each token with the appropriate content. Here is our current instruction: Instruction. OpenAI GPT 4o (I)Forecasting Multi-Modal Integration Q:· A: · Q: So, it‘s a small robot that can navigate through a maze using sensors. Isn’t that cool? <|image|>X<|image_end|> A:· · Q:It uses infrared sensors to detect obstacles and navigate through the maze. <|code|>X<| code_end|> A:· Q:· Instruction WithMulti- modalTokens More...... ToolkitforGenerating Multi-ModalContent Q: · A:· Multi-Round Multi-Modal Instruction More...... (IV)GeneratingMulti-ModalContent Q: · A:· · Google Map Google Search Meta MusicGen OpenAI DALL ·E 3 OpenAI GPT-4o Apple Emoji Archive Figure 3: Procedure of data construction. Stage (I) prepares diverse events by random topic pairs. Stage (I) leverages these events to produce multi-round instructions. Stage (I) forecasts eight types of multimodal indicator tokens and integrates these into instructions. Stage (IV) uses a toolkit to generate multimodal content. incorporate the relevant indicators. Beyond the base<language>modality, we fore- cast indicators for eight modalities:<image>, <music>,<emoji>,<code>,<map>,<link>, <math>, and<QR code>. Additionally, we gen- erate prompts that describe the corresponding mul- timodal content, as shown in Figure 3 (I). For in- stance, we generate prompts for GPT-4o (OpenAI, 2024) for modalities such as<image>,<code>, and <math>. For<map>and<link>, we develop search queries intended for use with Google Search and Google Maps. Stage (IV): Generating Multimodal ContentIn the final stage, we develop an advanced Toolkit for Generating Multimodal Content. This toolkit in- cludes GPT-4o (OpenAI, 2024), Google Search, Google Maps, DALL·E 3, MusicGen (Copet et al., 2023), and the Apple Emoji Archive to assist in pro- ducing multimodal content. We leverage the gener- ated prompts/content/queries to drive this toolkit, enabling the creation of multimodal content. 3.2 Data Format and Statistics Each UniDataset sample is generated by mapping a contextual triple to a multimodalinstruction that contains nine typed slots, one per modality (Eq.(1)). Slots are filled by dedicated genera- tors (Appendix A.2) conditioned on a modality- specific cue (a<prompt>,<label>,<query>, or <content>string), while the<language>slot car- ries the dialogue text itself. <event>, <emotion>, <occasion> Generate −→ instruction                        → <language> <prompt>→ <image> <prompt>→ <music> <label>→ <emoji> <prompt>→ <code> <query>→ <map> <query>→ <link> <prompt>→ <math> <content>→ <QR code>. (1) 4 Methodology Previous works, such as Multimodal Self- Instruct (Zhang et al., 2024) and VIGC (Wang et al., 2024a), typically support only limited multimodal generation or understanding capabilities, which do not satisfy users’ practical needs. In real-world applications, users often prefer to provide only one or a few simple requirements to generate large vol- umes of high-quality data. OverviewTo address the aforementioned issues, we propose UniData, a pipeline for generating multimodal instructions. Initially, based on user- provided requirements, we develop an event gener- ator that expands on the limited inputs to create di- verse events in Section 4.1. Next, to generate mul- timodal instructions, we construct a large model for multimodal understanding and generation in Section 4.2. Finally, to mitigate potential biases during long-term multi-round instruction genera- tion, we correct the instruction flow to filter out unreasonable data in Section 4.3. 4.1 Expanding Diverse Events Typically, users seek to generate one high-quality instruction dataset by providing only a few key- word requirements (K), such that (K→ P N n=1 I n ). However, if we treat oneKas a single<event>(T), it becomes challenging to produce a diverse set of instructions. Therefore, we first need to expand the variety of T, so we build the Diverse Event Generator (DEG) to expand diverse events, as shown in Figure 4 and Eq. (2), as N X n=1 I n ←T 1 , T 2 , . . . , T N = DEG(K⊕ S),(2) where⊕is the concatenation operator.The setT 1 , T 2 ,..., T N denotes the collection of <events>, each corresponding to a distinct instruction(I n ). The termSrepresents stochas- tic factors, such as random emotion proportions, random occasions, and other random seeds, which collectively enhance the capability ofDEG(·)to generate a broader spectrum of diverse events. 4.2 Generating Multimodal Instructions After obtaining diverse<events>, the next step is to generate multimodal instructions. Inspired by previous any-to-any models (Wu et al., 2024; Xie et al., 2025), we develop the Multimodal In- struction Generator (MIG), a text-centred genera- tor coupled with modality-specific realisation tools that can both understand and generate multimodal content, as shown in Figure 5 and Eq. (3), I M n = MIG(T n ⊕ M−1 X m=1 I m n ),(3) whereMIG(·)takes as input one eventT n and previously generated multimodal instructions ( P M−1 m=1 I m n ), and predicts the next-round multi- modal instruction (I M n ). Multimodal TokenizationTo generate the next- round instructions (I M n ),MIGmust understand the multimodal instruction from previous rounds ( P M−1 m=1 I m n ). Thus, it is necessary to convert the previous multimodal instructions into token embed- dings within a unified feature space. Given that our training dataset, UniDataset, com- prises nine distinct modalities, our tokenizing strat- Random Seeds Emotions Large Language Model (LLaMA-3) Key Words Diverse Tokens Diverse Events User Occasions 푵× 푵× Stochastic Factors Figure 4: Pipeline of Diverse Event Generator (DEG). Users can generate diverse events by inputting their needs. egy (F(·)) is shown as Eq. (4), F(I) =   f text [I L , C(I E ⊕ I C ⊕ I MP ⊕ I LK ⊕ I MT ⊕ I Q )] f img (I I ) f mus (I M )   , (4) wheref x (·),x ∈ text, img, musrepre- sentthetokenizerandembeddinglay- er/projectionfordifferentmodalities (I x ,x∈L, I, M, E, C, MP, LK, MT, Q). For modalities that are not inherently linguis- tic but can be represented as text (I x ,x ∈ E, C, MP, LK, MT, Q), we can easily convert these into textual representations (prompt/label/- query/content) by the formatter (C(·)). Then, these modalities can be combined with the language modality (I L ) and tokenized directly by using the default LLaMA Tokenizer (Grattafiori et al., 2024) (f text (·)). The tokenized outputs are subsequently passed through a pre-trained embedding layer to generate token embeddings. For<image>(I I ), we utilize the visual tok- enizer and projection (f img (·)) from Ovis-1.5 (Lu et al., 2024b) to convert images into visual em- beddings. For<music>(I M ), drawing on previ- ous research (Gong et al., 2023; Chu et al., 2023; Du et al., 2023; Verma, 2024), we integrate a pre- trained Whisper (Radford et al., 2023) as the music tokenizer, followed by a Multi-Layer Perceptron (MLP) that projects into the desired embedding space. Multimodal Indicator To distinguish different modalities, we introduce new special tokens to in- dicate the beginning and end of each modality. As shown in Figure 5, the start of a modality is indi- cated by <|modality|>, where modality∈image, music, emoji, code, map, link, math, qrcode, (5) and the end of the modality is indicated by <|modality_end|>.This strategy allows the Multi-Modality LargeLanguageModel(LLaMA-3) ퟏEvent Stable Diffusion A Q MusicGen 퐌RoundMulti-ModalInstruction · Image Music Emoji Code Link Map QRCode Generator (qrcode) A Q Visual Tokenizer Projection Music Tokenizer Projection TextTokenizerEmbeddingLayer ToLabel/Query/Content Round PreviousMulti-RoundsInstructions 퐌−ퟏ <|modality_end|> TokensforDifferentModalities <|modality|> Math Emoji Archive Retrieva l Figure 5: Pipeline of Multimodal Instruction Generator (MIG). With just one event provided by the user,MIG can iteratively generate multi-round multimodal instructions based on the context. model to identify the position of multimodal in- sertion. Architecture of Large Language Model After aligning the multimodal signals into a unified em- bedding space, we select LLaMA-3 (Grattafiori et al., 2024) as the main backbone of our pipeline. We also use the causal attention mechanism to se- quentially predict the next token. Multimodal Generation To generate multi- modal instructions, we first need to establish multi- modal token labels during training. Although most modalities can be represented as text during to- kenization, the image and music modalities lack corresponding text representations. However, we find that during data construction (as described in Eq.(1)), we obtain<prompts>(P I andP M ) for generating images and music. Therefore, we can use these prompts as tokens to represent these two modalities discretely. Eq.(6)shows the ground truth label (V), V =V L , (V E , V C , V MP , V LK , V MT , V Q ), V I , V M , where V I = f text (P I ),V M = f text (P M ), (6) andf text (·)is the tokenizer of LLaMA- 3 (Grattafiori et al., 2024) andV x ,x ∈ L, I, M, E, C, MP, LK, MT, Qare the labels of different modalities. Our goal is to predict the next token (V k ) given the previous tokens (V i k−1 i=1 ) and the model parameters (Θ), as Eq. (7), L =− X k log p(V k |V i k−1 i=1 ; Θ).(7) Multimodal DetokenizationWe use the default LLaMA-3 text detokenizer (Grattafiori et al., 2024) to convert all modalities into their text represen- tations. For<code>,<map>,<link>, and<math>, they can be directly detokenized into their original information. For<image>, we use Stable Diffusion 3 (Esser et al., 2024) to transform the text into the corresponding image. For<music>, we employ MusicGen (Copet et al., 2023) to convert the text into the corresponding music. For<emoji>, we perform a similarity search of the output text to retrieve the corresponding emoji from the emoji archive. For<QR code>, we can directly convert its content into a QR code by qrcode 1 . 4.3 Correcting Instruction Flow Although now we can generate multimodal instruc- tions byMIG, during inference, it may also en- counter common issues like repetitive generation or hallucinations (Xu et al., 2022; Huang et al., 2024). Therefore, cleaning and filtering data dur- ing long-term inference is crucial for maintaining data quality. We design an inference chain in Fig- ure 6. We compare the newly generated one-round instruction (I M n ) against the previousk-round in- structions ( P M−1 m=M−k I m n ) and apply the following three rules: 1.Repetitive Generation: If the new instruction is close to the previous ones, increase the REPE- TITION PENALTY (↑) to reduce repetition. 2.Irrelevant Generation: If the new instruction 1 https://pypi.org/project/qrcode/ Event QA: M QA: M-1 ...... Previous-rounds Instructions Calculating Deviation 퐑<훼 ! 퐑>훼 " 훼 " >퐑>훼 ! Changing Parameter and Regenerating Tr a n s f e r Rules Temperature ↑ Repetition Penalty ↑ ① ③ ② To-Next ① ③ ① ③ ② Figure 6: Inference chain for correcting instruction flow. is unrelated to the previous ones, raise the TEM- PERATURE (↑) to introduce randomness, thereby sampling more alternatives. 3.High-Quality Generation: If the instruction is neither repetitive nor irrelevant, we regard it as high-quality and proceed to the generation of the next round. To assess deviation, we use cosine similarity, as Eq. (8), R = 1 k M−1 X m=M−k cos_s(I M n , I m n ) = 1 k M−1 X m=M−k I M n · I m n |I M n ||I m n | . (8) IfR < α 1 , it indicates “Irrelevant Generation”. If R > α 2 , it indicates “Repetitive Generation”. If α 2 > R > α 1 , it indicates “High-Quality Genera- tion”. Empirically, we currently setα 1 = 0.05and α 2 = 0.4to minimize errors. Users can also adjust these values dynamically to maintain instruction quality and relevance based on their needs. 5 Experiments In this section, we compare our instruction gen- eration pipeline with others to evaluate instruc- tion quality (see Section 5.1). Besides, we as- sess whether the generated instructions can im- prove multimodal generation and understanding performance after fine-tuning (see Section 5.2). Fi- nally, we demonstrate the effectiveness of expand- ing events and correcting instruction flow (see Sec- tion 5.3). Furthermore, we conduct a user study to assess the quality of the generated instructions, demon- strate the adaptability and balance of multimodal content to support different needs, validate perfor- mance on out-of-domain data, and provide addi- tional qualitative results in the Appendix. Training & Testing During the pre-training phase, we adhere to the training strategy and dataset outlined in Ovis-1.5 (Lu et al., 2024b), enabling the model to acquire initial multimodal understand- ing ability. In the fine-tuning phase, we keep the parameters of the visual and music tokenizers fixed and randomly select90%of UniDataset for model fine-tuning, reserving the remaining10%for test- ing. 5.1 Instruction Quality Evaluation Baselines We select two representative instruc- tion generation frameworks, Self-Instruct (Wang et al., 2023) and VIGC (Wang et al., 2024a), for comparison on our test data. In Self-Instruct (Wang et al., 2023), due to the unavailability of GPT-3, we replace it with the latest GPT-4 (OpenAI, 2024). A discussion of more recent instruction-data gener- ation methods and how they compare in scope is provided in Appendix C.7. EvaluationMetrics Followingprevious work (Li et al., 2023b; Tang et al., 2024), we use GPT-Guided metrics to evaluate the quality differ- ences between generated results and references. Our goal is not to precisely match the ground truth but to produce high-quality instructions. We focus on four key metrics:1Reasonableness - assesses if the instruction is logical and appropriate;2 Clarity - ensures the instructions are clear and unambiguous;3Detail - evaluates whether the instructions include more detailed information;4 Relevance - checks that the content is related to the desired outcome. Performance AnalysisTable 1 presents the quan- titative results of our data quality. UniData sur- passes both baselines on all four dimensions, with the largest margin on Clarity and Relevance. Fur- thermore, UniData supports more rounds of instruc- tion generation and can produce rich multimodal content within instructions. Figure 7 shows the qualitative performance in data quality. Compared to other baselines that are limited to generating textual content, UniData can generate high-quality multimodal instructions. 5.2 Post Fine-Tuning Assessment Referring to previous research (Wang et al., 2024a, 2023), we also evaluate the effectiveness of the generated multimodal instructions (+I UniData ) by fine-tuning other multimodal models. Table 1: Quantitative performance in data quality. Red indicates the best performance. (#Others includes all other modalities (#Emoji, #Code, #Math, #Map, #Link, #QR Code)). MethodBackbones GPT-Guided Metrics(↑) (OpenAI, 2024)Avg. #Multimodalities(↑) Avg. #Rounds Reasonableness Clarity Detail Relevance#Image #Music #Others Self-Instruct (Wang et al., 2023)GPT-4 (OpenAI, 2024)0.6240.4440.5940.520---≈ 3 VIGC (Wang et al., 2024a)Vicuna 7B (Chiang et al., 2023)0.4600.3590.5340.569---≈ 1 Ours (UniData)LLaMA-3 7B (Grattafiori et al., 2024)0.6610.5270.6670.6821.920.747.37≈ 17.5 Table 2: Improvement of music understanding. BLEU (↑) BLEU-4 (↑) METEOR (↑) ROUGE (↑) BERT (↑) Baseline0.26750.20190.32320.33760.8932 Baseline +I UniData 0.26810.20760.33300.34960.8936 Table 3: Music generation. FAD (↓) CLAP (↑) KL (↓) Baseline16.690.31382.1358 Baseline +I UniData 16.510.31922.1038 Table 4: Improvement on different modalities. ModalityCode(↑)Math(↑)Text-based(↑) Reasoning(↑) BenchmarkMBPPMathVistaMMLU-ProMathVision Baseline20.859.420.312.8 Baseline +I UniData 21.360.121.913.0 Table 5: Image understanding/generation. Image Understanding Image Generation (↑) Single (↑) Multi (↑) Interleaved (↑) Baseline26.19%37.73%19.42%24.43% Baseline +I UniData 27.15%39.09%35.25%25.90% Table 6: Improvement on visual understanding. BenchmarkMMMU(↑) ChartQA(↑) TextVQA(↑) Baseline35.817.458.9 Baseline +I UniData 36.417.960.1 Table 7: Quantitative performance in ablation study. Reasonableness(↑) Clarity(↑) Detail(↑) Relevance(↑)#Ins. w/o Expansion0.5570.5180.6070.550k w/o Correction0.6480.4890.6100.654N × k Ours0.6610.5270.6670.682N × k VisualUnderstanding Weutilizea suite of benchmarks (MMMU (Yue et al., 2024), ChartQA (Masry et al., 2022), and TextVQA (Singh et al., 2019)) to evaluate the effec- tiveness of our generated data in visual reasoning tasks. As detailed in Table 6, the integration of our data improves the performance of Ovis2-1B (Lu et al., 2024b) in these benchmarks. Image Understanding and GenerationWe eval- uate image understanding and generation capabili- ties using the SEED-X model (Ge et al., 2024) and the SEED-Bench-2 benchmark (Li et al., 2023a). Our data improves both Image-to-Text and Text- to-Image tasks. As shown in Table 5, the gain is largest on the interleaved setting (19.42% → 35.25% ), with smaller gains on the single- and multi-image settings. Additionally, in the domain of image generation, our dataset helps the model produce relevant images based on textual inputs. Music Understanding and Generation We uti- lize the MuMu-LLaMA model (Liu et al., 2024a) to assess advancements in music understanding and generation. As illustrated in Tables 2 and 3, our generated instructions improve both domains, although the margins are small. In the realm of mu- sic understanding (Table 2), all five metrics move in the expected direction, with the clearest gains on METEOR and ROUGE. In music generation (Table 3), the fine-tuned model exhibits decreased FAD and KL scores alongside an elevated CLAP score, reflecting more coherent and high-quality music synthesis from textual prompts. Code GenerationWe evaluate the impact of our generated data on code modality by enhancing the code generation capabilities of a large language model, LLaMA 2 (Touvron et al., 2023), using the Mostly Basic Python Programming (MBPP) Benchmark (Austin et al., 2021). As presented in Table 4, our instructions improve performance. Mathematical ProblemTo evaluate the efficacy of our generated instructions in enhancing complex reasoning and mathematical problem-solving ca- pabilities, we benchmark the Ovis2-1B (Lu et al., 2024b) on the MathVista (Lu et al., 2024a) and MathVision (Wang et al., 2024b) datasets. In Ta- ble 4, our instructions improve performance on both mathematical benchmarks. The gains are con- sistent in direction across the two, though modest in magnitude. Other Text-based Modalities To confirm the versatility of our data across various text-based modalities, including language, QR code, link, and map, we assessed their performance using a general language-based benchmark, MMLU-Pro (Wang et al., 2024c). As shown in Table 4, adding our data raises MMLU-Pro accuracy from20.3to21.9, indicating that the text-based modalities also trans- fer to a general language benchmark. Event ---------------------------------------------------------------------------------- Two friends discuss their favorite recipes, share cooking tips, and plan a weekend cooking session together. VIGC ---------------------------------------------------------------------------------- Q: Two friends discuss their favorite recipes, share cooking tips, and plan a weekend cooking session together. A: By sharing their favorite recipes, cooking tips, and planning a weekend cooking session together, the friends can bond over their passion for cooking and enjoy the process of creating a delicious meal together. Self-Instruct ---------------------------------------------------------------------------------- Q: Hey, I've been looking for new recipes to try out. Do you have any favorites? A: Absolutely! I'm a big fan of homemade lasagna. It's a bit time-consuming, but it's totally worth it. What about you? Q: I love making chicken curry. It‘s a family recipe, and I’ve perfected it over the years. By the way, anytips for making lasagna easier. A: Sure! One tip is to use no-boil lasagna noodles. They save a lot of time and taste just as good. Also, try mixing ricotta with some spinach for added flavor and nutrition. · (More) Ours ---------------------------------------------------------------------------------- Q: Hey, I was just thinking about our cooking session this weekend. Do you have any recipe ideas in mind? A: Yeah, I was thinking we could make homemade pasta. · (More) Q: Thanks for the tip. What about sauce? Do you make your own tomato sauce or use pre-made? A: I prefer making my own. Fresh tomatoes add such a good flavor. <emoji> Q: I use jarred sauce, but I can see how fresh would be better. What your sauce like? · (More) A: It is simple tomatoes, olive oil, onions, and a sprinkle of sugar to balance the acidity. <music> · (More) A: I want to try making pasta with beetroot. I hear it really good for you. .<image> · (More) Italian Cooking Background Music Figure 7: Qualitative comparison in data quality. Event 1: A group of friends expressed concerns about the impact on small businesses and local economies. Event 2: A group of gardening enthusiasts discussed concerns about product quality. Event 3: A group of gardening enthusiasts shared tips on sourcing reliable online retailers and the importance of customer reviews. Keywords: e-commerce and gardening Figure 8: (left) Difference distribution of the events in three random generations. (right) Example of diverse events based on the same keywords. 5.3 Ablation Study Effectiveness of Expanding Diverse EventsTo demonstrate the effectiveness of expanding events, we randomly expand three events for each keyword on the test data to observe whether these events differ from one another (diversity). As shown in Figure 8 (left), the TF-IDF (Ramos et al., 2003) feature distributions across three generation runs (on a random subset) show considerable diversity. In Figure 8 (right), we also provide an example showing that different events exhibit distinct con- tent (in highlighted text). In the w/o Expansion setting of Table 7, the generation of instructions relies solely on the few keywords provided by the user. Thus, the generation pipeline lacks the nec- essary enriched prompt information, which leads to a decline in the quality of generated instructions. Previous Rounds Instruction: · (More) Let's make a difference and ensure that pets are treated with the love and respect they deserve. A: Agreed. Together, we can create a better world for animals and strengthen the bond between humans and their pets. <emoji> Repetitive Generation -------------- <emoji> · (More) <emoji> Irrelevant Generation -------------- <code> ```parm``` OR “null” After Correction (High Quality Generation) --------------------------------------- Q: What's your favorite part about having a pet? A: For me, it‘s the unconditional love and companionship they provide. <image> Figure 9: Example of flow error correction. Additionally, the number of instructions that can be generated is substantially reduced. Effectiveness of Correcting Instruction Flow To demonstrate the effectiveness of correcting in- struction flow, we run inference twice, with and without the correction, as shown in Table 7 and Fig- ure 9. Table 7 shows that after correction, our data quality improves on all four dimensions. Figure 9 provides an example of the repetitive generation problem (same<emojis>) and irrelevant genera- tion problem (<code>or “null”) in multimodal con- tent. After correction, instruction generation can effectively avoid these errors. 6 Conclusion This paper presents UniData, a universal pipeline for multimodal instruction generation.As a fundamental contribution, UniData establishes a framework for scalable data acquisition critical to MLLMs. By delivering high-quality, diverse, and cost-effective instruction sets, UniData accelerates progress in multimodal learning paradigms and their downstream applications. Limitations UniData is a text-centered orchestration pipeline rather than a native any-to-any model, so per- modality quality is bounded by the external tools that realize each modality. Long-horizon structured reasoning is handled only by simple correction rules, and the nine-modality coverage still omits video, 3D, motion, tables, biological data, and spo- ken input. Finally, UniData inherits biases from its LLM backbones, search APIs, and modality gener- ators; these may enter UniDataset and be amplified downstream, and systematic bias audits remain fu- ture work. References Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.https://lmsys.org/ blog/2023-03-30-vicuna/. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio- language models. Preprint, arXiv:2311.07919. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music gen- eration. In Thirty-seventh Conference on Neural In- formation Processing Systems. Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zi- chong Yang, Kuei-Da Liao, and 1 others. 2024. A survey on multimodal large language models for au- tonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, pages 958–979. Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. 2025. M- IFEngine: Towards multimodal instruction follow- ing. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1099– 1109. Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, Chang Zhou, Zhijie Yan, and Shiliang Zhang. 2023. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. Preprint, arXiv:2310.04673. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Do- minik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. 2024. Scaling rectified flow transform- ers for high-resolution image synthesis. In Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12606–12633. PMLR. Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and Kai Chen. 2024. MMBench-Video: A long-form multi-shot bench- mark for holistic video understanding. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. Yukang Feng, Jianwen Sun, Chuanhao Li, Zizhen Li, Jiaxin Ai, Fanrui Zhang, Yifan Chang, Sizhuo Zhou, Shenglin Zhang, Yu Dai, and Kaipeng Zhang. 2026. A high quality dataset and reliable evaluation for interleaved image-text generation. In International Conference on Learning Representations (ICLR). Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. 2024. Seed-x: Multimodal models with unified multi- granularity comprehension and generation. arXiv preprint arXiv:2404.14396. Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. 2023. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. Jacob Hansen, Wei Lin, Junmo Kang, Muhammad Je- hanzeb Mirza, Hongyin Luo, Rogerio Feris, Alan Ritter, James Glass, and Leonid Karlinsky. 2025. Instructify: Demystifying metadata to visual in- struction tuning data conversion. arXiv preprint arXiv:2505.18115. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. A survey on hallucination in large lan- guage models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems. Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2023a. Seed- bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092. Guoliang Li, Yudian Zheng, Ju Fan, Jiannan Wang, and Reynold Cheng. 2017. Crowdsourced data manage- ment: Overview and challenges. In Proceedings of the 2017 ACM international conference on Manage- ment of Data, pages 1711–1716. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b. Videochat: Chat-centric video un- derstanding. arXiv preprint arXiv:2305.06355. Luxi Li, Yuchen Li, Xiaotong Zhang, Yuhang He, Jian- jian Yang, Bin Tian, Yunfeng Ai, Lingxi Li, Andreas Nüchter, and Zhe Xuanyuan. 2024a. Embodied in- telligence in mining: Leveraging multi-modal large language model for autonomous driving in mines. IEEE Transactions on Intelligent Vehicles. Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, and Qing Li. 2024b. FIRE: A dataset for feedback integration and refinement evaluation of multimodal models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench- marks Track. Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. 2024. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Ar- tificial Intelligence and Control Engineering, CAICE ’24, page 405–409, New York, NY, USA. Association for Computing Machinery. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36. Shansong Liu, Atin Sakkeer Hussain, Qilong Wu, Chenshuo Sun, and Ying Shan. 2024a.Mumu- llama: Multi-modal music understanding and gen- eration via large language models. arXiv preprint arXiv:2412.06660. Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024b. MMBench: Is your multi-modal model an all-around player? In Computer Vision – ECCV 2024, pages 216–233. Springer. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun- yuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024a. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts.In Inter- national Conference on Learning Representations (ICLR). Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024b. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263– 2279. Association for Computational Linguistics. Henning Müller and Devrim Unay. 2017. Retrieval from and understanding of large-scale multi-modal medical datasets: a review. IEEE transactions on multimedia, 19(9):2093–2104. OpenAI. 2024.Gpt-4o system card.Preprint, arXiv:2410.21276. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak su- pervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518. PMLR. Juan Ramos and 1 others. 2003. Using tf-idf to deter- mine word relevance in document queries. In Pro- ceedings of the first instructional conference on ma- chine learning, volume 242, pages 29–48. Citeseer. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards VQA models that can read. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8317–8326. Tomáš Sou ˇ cek, Dima Damen, Michael Wray, Ivan Laptev, and Josef Sivic. 2024. Genhowto: Learning to generate actions and state transformations from instructional videos. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR). Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, and Ying-Cong Chen. 2024. HAWK: Learn- ing to understand open-world video anomalies. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 139751–139785. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 oth- ers. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. Prateek Verma. 2024. Whisper-gpt: A hybrid rep- resentation audio large language model. Preprint, arXiv:2412.11449. Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, and Conghui He. 2024a. VIGC: Visual instruction generation and correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5309–5317. Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024b. Measuring multi- modal mathematical reasoning with MATH-Vision dataset. In Advances in Neural Information Process- ing Systems (NeurIPS) Datasets and Benchmarks Track. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508. Association for Computational Linguistics. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024c. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat- Seng Chua. 2024. Next-gpt: Any-to-any multimodal llm. In Proceedings of the International Conference on Machine Learning, pages 53366–53397. Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. 2025. Show-o: One single transformer to unify multimodal understanding and generation. In Inter- national Conference on Learning Representations (ICLR). Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. In Advances in Neural Information Pro- cessing Systems. Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chen- liang Li, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. 2023. mplug-docowl: Modularized multi- modal large language model for document under- standing. Preprint, arXiv:2307.02499. Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, Jing Shao, and Wanli Ouyang. 2023. LAMM: Language-assisted multi- modal instruction-tuning dataset, framework, and benchmark. In Advances in Neural Information Pro- cessing Systems (NeurIPS) Datasets and Benchmarks Track. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal un- derstanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 9556– 9567. Letian Zhang, Quan Cui, Bingchen Zhao, and Cheng Yang. 2025. Oasis: One image is all you need for multimodal instruction data synthesis. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3542–3551. Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, and Yueting Zhuang. 2024. Multimodal self-instruct: Synthetic abstract image and visual reasoning instruction us- ing language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 19228–19252. Association for Computational Linguistics. Overview of the Appendix This appendix is organized as follows: •Dataset construction (Appendix A): the topic pool that seeds generation, the external multi- modal toolkit, dataset diversity, bias-mitigation procedures, and modality extensibility. •Methodology (Appendix B): a worked example of the instruction-flow correction step and an analysis of why UniData improves over a vanilla GPT-4o pipeline. •Experiments (Appendix C): training and infer- ence recipes, evaluation prompts, a user study, out-of-distribution results, and ablations on mul- timodal balance and user-conditioned generation. •Additional analyses: a scope comparison with recent instruction-data pipelines (Appendix C.7). •Future work (Appendix D) and additional quali- tative examples (Figures 10–13). A Dataset Construction A.1 Topic Pool We curate a topic pool of 90 domains spanning everyday life, science and engineering, business, the arts, and the humanities (Table 8). The breadth of this pool is the primary driver of UniDataset’s content diversity, as it determines the universe of scenarios from which subsequent events are drawn. For each instruction sample we draw two topics Q 1 ,Q 2 uniformly at random and merge them into a composite theme, ⟨topic⟩ = Q 1 ∩ Q 2 .(9) Intersecting unrelated domains (for instance, cook- ing∩chemistry or music∩marketing) introduces controlled stochasticity into the pipeline and yields interdisciplinary prompts that are difficult to obtain from any single-domain source. A.2 Multimodal Toolkit To realize the any-to-any generation interface de- scribed in the main paper, UniData orchestrates a suite of off-the-shelf tools, each responsible for a single modality. Table 9 summarizes every compo- nent, its target modality, its source, and the specific role it plays in the pipeline. We deliberately favor publicly available APIs and open-source models so that the pipeline can be reproduced and extended by the community. A.3 Data Diversity Table 10 compares UniDataset with widely-used multimodal instruction datasets along three or- thogonal axes: (i) the number of dialogue rounds per sample, (i) the count of distinct input/out- put modalities, and (i) the overall corpus size. UniDataset supports nine modalities on both the input and the output side, the largest modality set in this comparison, and provides on average17.5 rounds per dialogue (vs. at most4.8for prior work). A.4 Human Oversight for Bias Mitigation To reduce the bias inherited from any single LLM, we insert lightweight human oversight at two care- fully chosen points in the pipeline. Topic curation. The 90 seed topics are hand- selected to balance category coverage and cultural breadth, preventing the corpus from skewing to- ward a narrow set of domains. Event filtering. After event expansion, annota- tors discard implausible events and remove near- duplicates. Cleaner events produce better down- stream prompts and, importantly, mitigate the prop- agation of LLM-specific stylistic biases into the final instructions. A.5 Modality Extensibility The pipeline is designed to absorb new modalities without architectural changes. Stage 3 (raw-instruction generation).The plan- ner emits modality-agnostic slots that are filled at realization time, so adding a modality beyond the current nine does not require changing the planner. Stage 4 (multimodal realization). New APIs (such as Text-to-Video or Text-to-3D models) can be plugged in via a thin adapter that maps modal- ity slots to API calls, with no change to upstream components. B Methodology Details B.1 Instruction-Flow Correction in Practice Correction is performed before multimodal realiza- tion. At this stage every multimodal slot is still rep- resented by its textual prompt rather than the ren- dered media, so the corrector can reason about con- tent, dependencies, and redundancy purely at the language level. A representative snippet is shown below. Table 8: The 90-domain topic pool used to seed UniDataset generation. At sampling time two domains are intersected (Eq. 9) to form a composite theme. e-commerceacademic discussions lifestylesportsmathematicsbusiness technologytravelshealthentertainmentarthistory cookingparentingfashionfinancepoliticsliterature gardeningastronomymusiceducationcomputer scienceprogramming designresearchinternshipfoodbeautyentrepreneurship startupsalgorithmbugsscholarITvision supervisionclassroomassignmentgamepsychologysocial media filmtheaterarchitecturephotographyenvironmentagriculture real estatemarketingeconomicsphilosophylawmedicine engineeringphysicsbiologyroboticscybersecurityartificial intelligence familyspace explorationclimate changeurban planningtransportationhospitality tourismfitnessrelationshipsholidayscharityDIY projects survival skillslifecartoonpetswildlifemarine biology geneticsneuroscienceanthropologysociologylinguisticsculture mythologyarchaeologyforensic sciencecriminologymilitarydiplomacy Table 9: External tools used by UniData during multimodal realization. Each tool is responsible for one modality, allowing the pipeline to be extended modularly. ToolModalitySourceRole in the pipeline GPT-4o (OpenAI, 2024)<language>OpenAI APIOrchestrator and language backbone; expands events, drafts dialogue, and vali- dates outputs. Google Search<link>https://w.google.comProvides up-to-date fac- tual snippets and reference URLs. Google Maps<map>https://w.google.com/mapsSupplies geographical ref- erences and routing infor- mation. DALL·E 3<image>https://openai.com/index/dall-e-3/Synthesizes photo-realistic and stylized images from text prompts. MusicGen (Copet et al., 2023)<music>Open-source modelGeneratesinstrumental clips from textual descrip- tions. Apple Emoji Archive<emoji>https://w.apple.com/Provides culturally famil- iar emoji symbols for affec- tive cues. Table 10: Comparison with current multimodal instruction datasets. Num is the number of distinct modality types; Class lists the modalities themselves. Legend: L:<language>; I:<image>; V:<video>; 3D:<point cloud>; LK:<link>; A:<audio>; M:<music>; E:<emoji>; C:<code>; MP:<map>; MT:<math>; Q:<QR code>. Dataset Input ModalityOutput Modality Avg. #Rounds#Instruction NumClassNumClass LLaVA (Liu et al., 2023)2L/I1L≈ 1.05K VideoChat (Li et al., 2023b)2L/V1L≈ 1.811K FIRE (Li et al., 2024b)2L/I1L≈ 1.0100K LAMM (Yin et al., 2023)3L/I/3D1L≈ 3.3196K mPLUG-DocOwl (Ye et al., 2023)4L/I/T/L1L-- GenHowTo (Sou ˇ cek et al., 2024)2L/I2L/I-- MosIT (Wu et al., 2024) 4L/I/A/V4L/I/A/V≈ 4.85K Ours (UniDataset)9L/I/M/E/C/MP/LK/MT/Q9L/I/M/E/C/MP/LK/MT/Q≈ 17.520K Example: textualized multimodal slot during correction Q: . . . A: This further stimulates economic growth by expanding markets and reducing food scarcity issues. <|image|> Prompt: farmland used for marketing cam- paigns. <|image_end|> Because the textual prompt is already available at this stage, the corrector can detect redundancy or topical drift without inspecting the final media, which keeps correction fast and cheap. A tighter feedback loop that conditions the corrector on the rendered media itself is left to future work. B.2 Why UniData Outperforms a Vanilla Pipeline Richer multimodal toolkit. UniData couples GPT-4o with an external toolkit (DALL·E for im- ages, MusicGen for audio, search APIs for facts), extending the effective modality range far beyond what a single language model can produce natively. Pipeline design. In Table 12, Self-Instruct runs on a GPT-4 backbone while UniData runs on LLaMA-3 7B, yet UniData scores higher on all four GPT-guided metrics. Under this compari- son the gain is attributable to event expansion and instruction-flow correction rather than to backbone capacity. C Experimental Details C.1 Training and Inference Training ofDEG. The Diverse Event Genera- tor (DEG) is fine-tuned on top of LLaMA-3 using ⟨topic⟩→⟨event⟩pairs harvested from UniDataset. Fine-tuning takes roughly 40 hours on a single NVIDIA A100 (80 GB). Training ofMIG. The Multimodal Instruction Generator (MIG) is fine-tuned on four NVIDIA A100 (80 GB) GPUs for roughly 40 hours. C.2 Evaluation Prompt and Metrics We define four GPT-4o-judged metrics (Reason- ableness, Clarity, Detail, and Relevance) and query the judge with the unified prompt shown below. Unified GPT-4o evaluation prompt Evaluate themetricof the multimodal instruction gen- erated, compared to the ground truth reference. Return a single score in [0, 1]. Example output: "score": 0.675. Definition of metric: explanation. The four metrics are operationalized as follows: •Reasonableness: logical consistency and factual accuracy with respect to the reference. •Clarity: readability, well-formedness, and unam- biguity of the generated text. •Detail: richness of information conveyed in each turn. •Relevance: topical adherence to the input con- text across the full dialogue. The evaluator is asked to score each metric inde- pendently so that absolute and relative comparisons can be made across the four dimensions. C.3 User Study To complement the automatic GPT-4o scores, we run a forced-choice user study. We recruit 15 evaluators from a mix of academic and indus- trial backgrounds, all with prior experience with large language models. Each evaluator inspects 50 samples and, for every one of the four eval- uation dimensions, selects which method among Self-Instruct (Wang et al., 2023), VIGC (Wang et al., 2024a), and UniData produced the best out- put. The results, reported in Table 11, show that UniData is preferred by a substantial margin on every dimension. Table 11: User study: percentage of samples (%) on which each method is selected as the best by 15 hu- man evaluators across 50 samples (per-dimension forced choice). Red indicates the best performance. MethodReason. (↑)Clarity (↑)Detail (↑)Relev. (↑) Self-Instruct (Wang et al., 2023)18.59.79.88.5 VIGC (Wang et al., 2024a)10.35.314.410.9 Ours (UniData)71.285.075.880.6 C.4 Out-of-Distribution Performance To probe generalization, we construct an out-of- distribution (OOD) test set of 100 events drawn from domains outside the seed topic pool (collected via web scraping followed by manual screening). Table 12 shows that UniData remains the strongest method on every metric while simultaneously pro- ducing visibly richer multimodal content (more images, music, and other modalities per dialogue) and longer interactions. C.5 Multimodal Distribution across Dialog Position A common failure mode of existing pipelines is concentrating multimodal content in the opening turns of a dialogue. Table 13 shows that UniDataset Table 12: Quality of generated instructions on the out-of-distribution (OOD) set. Bold marks the best score in each column. #Others aggregates the emoji, code, math, map, link, and QR-code modalities. MethodBackbones GPT-Guided Metrics(↑)Avg. #Multimodalities(↑) Avg. #Rounds Reasonableness Clarity Detail Relevance#Image #Music #Others Self-Instruct (Wang et al., 2023)GPT-4 (OpenAI, 2024)0.7350.4750.6080.580---≈ 3 VIGC (Wang et al., 2024a)Vicuna 7B (Chiang et al., 2023)0.7140.5070.7320.737---≈ 1 Ours (UniData)LLaMA-3 7B (Grattafiori et al., 2024)0.7520.7010.8080.8451.040.273.82≈ 10 instead distributes multimodal content nearly uni- formly across the entire dialogue (the five normal- ized buckets each carry roughly 20 % of the mul- timodal mass). The qualitative examples in Fig- ures 11–13 corroborate this: modalities appear con- sistently in early, middle, and late rounds. Table 13: Fraction of multimodal content (%) as a func- tion of the normalized position within a dialogue. Posi- tion is binned into five equal-width buckets. Position in dialogueMultimodal frequency (%) 0 – 20 %20.5 20 – 40 % 19.4 40 – 60 %19.5 60 – 80 % 21.3 80 – 100 %19.3 C.6 Adaptation to User-Specified Modalities When the user steers the pipeline toward a partic- ular modality, the generated content shifts accord- ingly (Table 14). For example, an image-focused input yields roughly15×more image content than the general setting, while a music-focused input boosts the music count by a similar factor. The orchestration layer therefore responds to the re- quested modality rather than defaulting to a fixed mix. Table 14: Average count of each modality per dialogue when the user specifies a desired modality at input time. Red indicates the targeted modality in each row. Input type#Image#Music#Others General1.920.747.37 Music-related1.1012.302.10 Image-related14.900.105.20 C.7 Comparison with Recent Instruction-Data Pipelines The two baselines in Table 1 (Self-Instruct and VIGC) are chosen because they represent the two dominant styles of universal instruction-data gen- eration: single-modality seed-based expansion and vision-conditioned instruction generation. For com- pleteness, we position UniData against several re- cent instruction-data pipelines that appeared along- side or after this work. • Oasis (Zhang et al., 2025) generates language- side instruction–response pairs from a single conditioning image; assistant-side output is text- only. • Instructify (Hansen et al., 2025) iteratively gen- erates multi-turn visual instruction conversations by aggregating over40source datasets; assistant- side output is again text-only. •M-IFEngine (Ding et al., 2025) produces constraint-rich image–instruction–answer triples for SFT and DPO; assistant-side output is text- only. •InterSyn (Feng et al., 2026) synthesizes single- and multi-turn interleaved image–text dialogues, with assistant-side output covering language and image. These pipelines target different source data, tasks, and evaluation protocols, so their paper- reported scores cannot be placed in a common “quality” column with those in Table 1 without rerunning every method under an identical pro- tocol. We therefore treat them as scope compar- isons rather than head-to-head numeric competitors. The distinguishing feature of UniData relative to these pipelines is the combination of symmetric nine-modality input/output coverage (rather than a language-only or language+image assistant side) and long multi-round interactions (17.5 rounds on average). Among the recent works listed, Inter- Syn (Feng et al., 2026) is closest in spirit but still limits assistant-side output to language and image. C.8 Additional Qualitative Results Figure 10 visualizes the output diversity of DEG: three independently sampled events for the same keyword pair differ substantially in content, tone, and target modality. Figures 11–13 present three end-to-end exam- ples produced by MIG. Each example interleaves several modalities (image, music, code, emoji) across a multi-round dialogue, illustrating both the modality breadth and the long-horizon coherence achieved by UniData. D Future Work We view UniData as a foundation for the automated production of multimodal training data, and iden- tify three concrete directions for follow-up work. Scaling backbones.Replace the LLaMA-3 back- bone of MIG with more recent and larger open- source models to characterize how multimodal in- struction quality scales with backbone capacity. Expanding modalities. Plug in tools for video, 3D, motion, and tabular generation so that UniDataset can extend beyond the current nine modalities, increasing applicability to embodied and scientific domains. User-conditioned generation. Tighten the con- trol loop so that UniData behaves as an interactive data factory: an end user provides their downstream specification, and UniData produces purpose-built data aligned with that deployment scenario. Events ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Two friends discuss their favorite cartoons and how they use supervision to ensure their kids watch age-appropriate shows. Keywords ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cartoon and supervision Events ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A group of students and a teacher discuss the impact of sports on academic performance. Students share personal experiences, highlighting how playing sports has improved their focus, teamwork skills, and stress levels. The teacher emphasizes the importance of balancing academics and sports, agreeing that physical activity can enhance overall learning outcomes. Keywords ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- sports and education Events ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Two friends discuss how studying the relationship between humans and their pets can provide insights into cultural practices,social structures, and human evolution. Keywords ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- pets and anthropology Events ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A father teaches his children how to build a shelter and start a fire in the woods, emphasizing the importance of these skills for survival and family bonding. Keywords ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- survival skills and family Events ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Two friends discuss how social media influences film promotion, noting that viral trends can boost box office sales, while also considering the downside of online criticism and the pressure on filmmakers to create content constantly. Keywords ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- social media and film Events ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Two parents discuss how understanding their children's emotions can improve their parenting skills. Keywords ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- parenting and biology Figure 10: Diverse event generation by DEG. For the same keyword pair, three independently sampled events differ substantially in content, scenario, and intended modality, illustrating the diversity that drives downstream instruction variety. Event ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Two parents discuss the benefits of supervising their children while watching cartoons, agreeing that it ensures content appropriateness and provides opportunities for educational conversations. Ours -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Q: Good afternoon, how was your day today? A: Good afternoon. It was quite eventful, thank you. I heard you wanted to discuss something about our children and their TV time? Q: Yes, I was thinking about how important it is for us to supervise what they are watching. What are your thoughts on this? A: I completely agree. It's crucial to ensure that the content is appropriate for their age. There are so many shows that aren'tsuitable for children. Q: Absolutely. I find it quite disheartening when they come across something they should not have seen. Have you had any similarexperiences? A: Yes, I was quite surprised and upset the first time it happened. It can be quite distressing. Q: Indeed. It makes me fearful that they might be exposed to harmful content. How do you feel about the potential risks? A: I share your concerns. The internet is vast, and it's easy to come across inappropriate material. We need to be vigilant. Q: That true. I also feel a bit sad that they might miss out on educational content. How can we balance supervision with allowing them to explore safely? A: It‘s a delicate balance. We can set specific times for supervised viewing and alsointroduce them to a variety of educational shows. <link> ------------------------------------------------------------------------------------------------ Search: List of recommended educational TV shows for children Result: 1.What's on these kids TV shows? Any recommendations? Movies... 2.Can you recommend me a good TV series that is suitable for... -Reddit 3.TV Reviews, Kids TV shows | Common Sense Media ------------------------------------------------------------------------------------------------ Q: That sounds like a good plan. I was disgusted to hear that some parents are so lax. Do you think stricter supervision is necessary? A: Yes, definitely. Stricter supervision ensures that we can control the content they watch and monitor their viewing habits. Q: I felt a bit of fear thinking about the long-term consequences if we're not vigilant. Do you think it will help in their development? A: Absolutely. It provides a safe environment for them to learn and grow without being exposed to negative influences. <emoji> Q: I'm glad we are on the same page. It makes me happy to know that we can make a positive impact. How do you think we should startimplementing this? A: We could start by creating a list of approved shows and setting up parental controls on their devices. <code> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ · (More) approved_shows= [“show”: “Sesame Street”, “age_group”: “3-5”, "show": "Dora the Explorer", "age_group": "3-5", "show": "Blue's Clues", "age_group": "3-5", "show": "The Magic School Bus", "age_group": "5-8", "show": "Rugrats", "age_group": "5-8", "show": "Hey Arnold!", "age_group": "5-8", "show": "Recess", "age_group": "5-8", "show": "The Simpsons", "age_group": "8-12", "show": "Batman: The Animated Series", "age_group": "8-12", "show": "Talespin", "age_group": "8-12"] · (More) setup_parental_controls('parental_controls.json', approved_shows) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Q: That a practical approach. Do you think we should also discuss the shows with them to understand their preferences better? A: Yes, I think that‘s essential. It will give us a better understanding of what content they are interested in and what we should avoid. <emoji> Q: I felt a bit sad thinking about the time commitment this will require. Do you think it will be manageable? A: It will be challenging, but I believe it worth it. We can take turns supervising or even set up a shared schedule. .<image> Q: I was surprised by how much thought this requires. Do you think it will be effective in keeping them engaged while still being educational? A: Yes, I do. It will keep them engaged while we ensure they are watching content that is beneficial for their development. Q: I feel a bit fearful about the potential pushback they might give. What should we do if they resist? .<music> A: We need to remain consistent and communicate clearly why it's important for their well-being. It might take some time, but persistence is key. Q: I was disgusted by the idea that some parents might prioritize their own convenience over their children‘s safety. How do we ensure we stay committed? A: We should remind ourselves of our primary goal is keeping them safe and educated. It a team effort, and we need to support each other. <emoji> Q: I felt a bit sad thinking about the potential for missed opportunities if we're too strict. How do we find the right balance? A: We should be flexible and open to adjustments based on what works best for our family. It's not a one-size-fits-all solution. <math> $$ S = \ TF \× (1 -\ D100) *How to balance screen time $$ Q: I was happy to hear that. Do you think it will also help in teaching them valuable life lessons? A: Definitely. We can use TV time as an opportunity to discuss various topics and teach them critical thinking skills. <link> ------------------------------------------------------------------------------------------------ Search: Effective ways to teach children through TV shows Result: 1.Better Not TV: What Parents Should Know and Do About TV for Kids 2.How to Effectively Integrate TV into Your Kids' Curriculum -LinkedIn 3.Show Your Child the Difference Between TV Shows & Real Life00 ------------------------------------------------------------------------------------------------ Q: Certainly. It more than just watching cartoons; it about creating a positive and educational experience. <emoji> A: Agreed. By supervising their viewing habits, we are giving them the best possible start in life. Q: I feel a bit relieved knowing we have a plan. Shall we start implementing it this evening? A: Yes, let do that. We can create the list of approved shows and set up parental controls. It will be a good start. <map> Location: Our home address Address: 1815 Avenue M, New York, NY 100090 · (More) Calming Background for Education Figure 11: End-to-end qualitative example (I): a multi-round, multimodal instruction produced by UniData. Image, music, and text are interleaved into a coherent narrative. Event ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A panel of experts discussed the impact of technological advancements in agriculture on global economic growth, highlighting increased crop yields, reduced costs, and potential job displacement. Ours -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Q: Thank you all for joining this important discussion on the impact of technological advancements in agriculture on global economic growth. To start, can you explain the benefits of increased crop yields that technology brings? A: Certainly. Increased crop yields due to technological advancements result in higher food production, which can lead to lower food prices and increased consumer incomes. This further stimulates economic growth by expanding markets and reducing food scarcity issues. <image> Q: That's a positive outlook. However, you mentioned potential job displacement. Can you elaborate on that? A: The integration of technology such as automated machinery and precision farming systems can lead to job displacement, particularly for manual labor jobs. This shift requires workers to upgrade their skills beyond merely agricultural tasks, which can be both challenging and costly. <link> ------------------------------------------------------------------------------------------------ Search: Research articles on job displacement due to automation Result: 1.Understanding how automation affects workers, jobs, and wages 2.AI's impact on jobs and work is a choice between two futures 3.(PDF) The Unintended Consequences of Automation: Separating... ------------------------------------------------------------------------------------------------ Q: Given the fear this might instill in workers, what measures can be taken to mitigate these risks? A: Education and training programs are crucial. Governments and private sectors must invest in retraining workers to adapt to higher-tech roles, such as data analysis, machinery maintenance, and administrative tasks. This transition period can be supported through subsidies and workforce development initiatives. <link> ------------------------------------------------------------------------------------------------ Search: linking to resources for workforce development programs Result: 1.Workforce Development Services -Employers' Training Resources 2.U.S. Department of Labor -WorkSource Texas 3.Workforce Resources -Region 18 Workforce Development Board ------------------------------------------------------------------------------------------------ Q: What about the environmental impact? Can technological advancements in agriculture help address sustainability concerns? A: Absolutely. Technologies like precision agriculture can reduce waste and conserve resources by optimizing crop management. This not only benefits the environment by minimizing excess fertilizer and pesticide use but also improves crop quality and yields sustainably. <map> Location: Location of a sustainable farming project using precision technology Title: Mightybytestechnology Address: 35 11th Ave, New York, NY 10011 Q: Finally, considering these mixed effects, do you believe the overall economic growth justified the investment in these technologies, or should there be a more balanced approach? A: While the potential economic growth from increased yields and efficiency is significant, it's essential to weigh these benefits against the social costs of job displacement and the need for robust support systems for affected workers. A balanced approach that includes investment in education, social safety nets, and sustainable practices would be ideal. <math> $$ \ ( \α \·G(T) -\β \·C(T) \ ) $$ Q: Thank you for your insights. This has been a thought-provoking discussion on the multifaceted impact of technological advancements in agriculture. A: Thank you for having us. It's crucial to continue these conversations to ensure that technological progress benefits all sectorsof society, not just the economic ones. <emoji> Figure 12: End-to-end qualitative example (I): a different topic pair, showing variation in dialogue length and modality mix while preserving topical coherence. Event ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A panel of experts discussed the integration of artificial intelligence in robotics, focusing on advancements in machine learning, ethical considerations, and potential impacts on various industries. They highlighted the importance of collaboration between technologists and ethicists to ensureresponsible development and deployment. Ours -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Q: Good evening, everyone. Let's begin our discussion on the integration of artificial intelligence in robotics. What do you think are the recent advancements in machine learning that are most impactful? A: One of the most impactful advancements is in deep learning and reinforcement learning. These technologies have significantly enhanced the ability of robots to learn and adapt to complex environments autonomously. Q: That sounds promising, but what about the ethical considerations? How do we ensure these advancements are used responsibly? A: Ethical considerations include ensuring transparency in AI decision-making processes and preventing biases. Collaboration with ethicists is crucial to create frameworks that guide responsible development and deployment of robotic systems. Q: Can you elaborate on the potential impacts of AI in various industries? A: AI can revolutionize industries by enhancing efficiency and decision-making accuracy. For example, in healthcare, AI-driven robots can assist in surgeries, providing precision and reducing risks. In the automotive industry, AI can improve automation and safety features in vehicles, reducing human error. Additionally, in the financial sector, AI can facilitate fraud detection, thus safeguarding customers.<image> Q: Those examples are compelling, but they also raise concerns. What are some of the risks associated with AI on a broader scale? A: Risks include job displacement due to automation and cybersecurity threats as AI systems become more sophisticated. It's essential to address these challenges through robust policies and continuous innovation in security measures. Q: Collaboration between technologists and ethicists is often mentioned as a solution. How can they effectively work together toaddress these challenges? A: Effective collaboration can be fostered through interdisciplinary teams, mutual respect, and open communication channels. It‘s important to integrate ethical considerations from the initial design phase to ensure sustainable and responsible AI applications. <link> ------------------------------------------------------------------------------------------------ Search: Best practices for interdisciplinary collaboration in AI and ethics Result: 1.A framework for motion towards human AI-Ethics at scale -ScienceDirect.com 2.Embedding worst-case scenarios into responsible artificial intelligence... 3.[PDF] Responsible Cost Ag_shareWEB downloadLevelsof abstraction... ------------------------------------------------------------------------------------------------ Q: Given all these developments, what role should government regulation play in the integration of AI in robotics? A: Government regulation should ensure that ethical guidelines are adhered to and that accountability mechanisms exist for misuse or unintended consequences. Regulations can also promote innovation by setting standards while adapting as technology evolves. Q: Are there any examples where government regulation has successfully balanced innovation and safety? A: The European Union's General Data Protection Regulation (GDPR) is one example. It has balanced privacy rights and innovation by implementing stringent data protection measures, which has largely been met with positive reception. Q: How do we address the concerns of those who are fearful or distrustful of AI? A: Addressing fears requires transparency and education. By demonstrating ethical practices and tangible benefits of AI, we can build trust and alleviate concerns. It‘s also important to involve stakeholders in decision-making processes. <music> Q: What about the potential for AI to perpetuate inequalities if not handled correctly? A: AI can perpetuate inequalities if access to technology remains limited to privileged groups or if it is used in ways that advantage certain demographics over others. Policies should ensure equitable access and unbiased applications. Q: It's a complex issue with no easy solutions. How can we stay committed to ethical AI development while rapidly advancing technology? A: Committing to ethical AI development requires ongoing vigilance, continuous education, and the integration of ethical principlesinto all stages of technology development. It also requires a shift in mindset where ethical implications are given equal priority with technical innovation. Q: In conclusion, what are the key takeaways from this discussion on AI integration in robotics? A: Key takeaways include the importance of machine learning advancements, the need for ethical considerations, the potential positive impacts across various industries, and the necessity for collaboration between technologists and ethicists to ensure responsible developmentand deployment of AI in robotics. <QR Code> Q: Thank you all for your insights. As we continue to work towards a responsible integration of AI in robotics, what final thoughtswould you like to share with our audience? A: It‘s an exciting journey with many possibilities. We must remain vigilant, collaborate openly, and prioritize ethical considerations. Together, we can harness AI’s potential to positively benefit society while minimizing risks. Q: Can you show me a visual representation of AI in robotics? A: Sure, here‘s an image of a robot using AI to navigate a complex environment. <image> Q: That's fascinating. Can you summarize the three main ethical concerns discussed? A: Yes, they focused on transparency, prevention of biases, and accountability. Q: What are some potential impacts of AI on the job market? A: AI can automate certain tasks, leading to job displacement in some sectors, but it can also create new opportunities in tech andAI- related fields. <link> ------------------------------------------------------------------------------------------------ Search: AI impact on job market Result: 1.AI Will Transform the Global Economy. Let's Make Sure It Benefits... 2.The Impact of Artificial Intelligence on Employment: Trends and... 3.The Impact of AI on the Job Market: Trends and Future Projections ------------------------------------------------------------------------------------------------ · (More) Q: What was the response to the GDPR regulation in the tech industry? A: Responses were mixed; while some welcomed it for protecting privacy, others expressed concerns about its potential to stifle innovation. <emoji> Q: Can you write a code snippet for educating AI ethics? A: Sure, here's a basic code snippet for educational purposes. <code> importrandom classAI_Ethics_Quiz: def__init__(self): self.questions=[ “question”: “What are some ethical concerns regarding AI development?”, “options”: [“Bias in AI”, “Job displacement”, “Autonomous weapons”, “Risk of misuse”], "answer": "Bias in AI", "question": "Which of the following is a risk associated with AI but not with human actions?", "options": ["Increased surveillance", "Improved efficiency", "Enhanced creativity", "Job displacement"], "answer": "Increased surveillance", "question": "What is the most important principle for developing AI?", "options": ["Maximize profit", "Ensure transparency", "Ensure fairness", "Ensure safety"], "answer": "Ensure fairness" ] defshuffle_questions(self): random.shuffle(self.questions) defask_question(self): print(“Welcome to the AI Ethics Quiz!”) fori, question inenumerate(self.questions): print(f“i+1: question[‘question’]”) forj, option inenumerate(question[‘options’]): print(f"j+1. option") answer =input("Your answer: ") print("Correct Answer:", question['answer']) print() defplay_quiz(): quiz =AI_Ethics_Quiz() quiz.shuffle_questions() quiz.ask_question() play_quiz() · (More) Background music to reduce fear and anxiety about AI Figure 13: End-to-end qualitative example (I): a long-horizon dialogue that interleaves text, image, code, and emoji content, illustrating UniData’s behavior on extended interactions.

Instant research discovery