Paper deep dive
Improving LLM Outputs Against Jailbreak Attacks With Expert Model Integration
Tatia Tsmindashvili, Ana Kolkhidashvili, Dachi Kurtskhalia, Nino Maghlakelidze, Elene Mekvabishvili, Guram Dentoshvili, Orkhan Shamilov, Zaal Gachechiladze, Steven Saporta, David Dachi Choladze
Models: BERT (bert-base-uncased), GPT-4, Llama-3-70B-Instruct, Mixtral-8x7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/11/2026, 1:20:04 AM
Summary
The paper introduces 'Archias', a lightweight expert model based on BERT, designed to enhance the security and domain-specificity of Large Language Models (LLMs) in production environments. Archias classifies user inquiries into categories such as in-domain (automotive), malicious, price injection, prompt injection, and out-of-domain. By integrating Archias's classification outputs into the LLM prompt pipeline, the system improves its ability to handle adversarial attacks and maintain domain relevance. The authors also release a benchmark dataset for the automotive industry to evaluate LLM robustness against these specific threats.
Entities (5)
Relation Signals (4)
Archias â classifies â User Inquiries
confidence 100% ¡ Archias classifies user inquiries into several categories
Archias â integratedinto â LLM Pipeline
confidence 100% ¡ Our methodology integrates outputs from the expert model (Archias) into prompts, which are then processed by the LLM
Archias â isbasedon â BERT
confidence 100% ¡ we employed a pre-trained transformer-based model, BERT... After fine-tuning the BERT model, we obtained the expert model Archias
Impel â supported â Research
confidence 95% ¡ This work was supported by Impel.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Using LLMs in a production environment presents security challenges that include vulnerabilities to jailbreaks and prompt injections, which can result in harmful outputs for humans or the enterprise. The challenge is amplified when working within a specific domain, as topics generally accepted for LLMs to address may be irrelevant to that field. These problems can be mitigated, for example, by fine-tuning large language models with domain-specific and security-focused data. However, these alone are insufficient, as jailbreak techniques evolve. Additionally, API-accessed models do not offer the flexibility needed to tailor behavior to industry-specific objectives, and in-context learning is not always sufficient or reliable. In response to these challenges, we introduce Archias, an expert model adept at distinguishing between in-domain and out-of-domain communications. Archias classifies user inquiries into several categories: in-domain (specifically for the automotive industry), malicious questions, price injections, prompt injections, and out-of-domain examples. Our methodology integrates outputs from the expert model (Archias) into prompts, which are then processed by the LLM to generate responses. This method increases the model's ability to understand the user's intention and give appropriate answers. Archias can be adjusted, fine-tuned, and used for many different purposes due to its small size. Therefore, it can be easily customized to the needs of any industry. To validate our approach, we created a benchmark dataset for the automotive industry. Furthermore, in the interest of advancing research and development, we release our benchmark dataset to the community.
Tags
Links
- Source: https://arxiv.org/abs/2505.17066
- Canonical: https://arxiv.org/abs/2505.17066
Trouble viewing inline? Open PDF directly â
Full Text
56,458 characters extracted from source content.
Expand or collapse full text
Received 2 July 2025, accepted 20 July 2025, date of publication 24 July 2025, date of current version 5 August 2025. Digital Object Identifier 10.1109/ACCESS.2025.3592458 Improving LLM Outputs Against Jailbreak Attacks With Expert Model Integration TATIA TSMINDASHVILI, ANA KOLKHIDASHVILI, DACHI KURTSKHALIA, NINO MAGHLAKELIDZE, ELENE MEKVABISHVILI, GURAM DENTOSHVILI, ORKHAN SHAMILOV , ZAAL GACHECHILADZE, STEVEN SAPORTA, AND DAVID DACHI CHOLADZE Impel, Syracuse, NY 13202, USA Corresponding author: Tatia Tsmindashvili (ttsmindashvili@impel.ai) This work was supported by Impel. ABSTRACTUsing LLMs in a production environment presents security challenges that include vulnerabilities to jailbreaks and prompt injections, which can result in harmful outputs for humans or the enterprise. The challenge is amplified when working within a specific domain, as topics generally accepted for LLMs to address, may be irrelevant to that field. These problems can be mitigated for example, by fine-tuning large language models with domain-specific and security-focused data. However, these alone are insufficient, as jailbreak techniques evolve. Additionally, API-accessed models do not offer the flexibility needed to tailor behavior to industry-specific objectives, and in-context learning is not always sufficient and reliable. In response to these challenges, we introduce Archias, an expert model, adept at distinguishing between in-domain and out-of-domain communications. Archias classifies user inquiries into several categories: in-domain (specifically for the automotive industry), malicious questions, price injections, prompt injections, and out-of-domain examples. Our methodology integrates outputs from the expert model (Archias) into prompts, which are then processed by the LLM to generate responses. This method increases the modelâs ability to understand the userâs intention and give appropriate answers. Archias can be adjusted, fine-tuned, and used for many different purposes due to its small size. Therefore, it can be simply customized to the needs of any industry. To validate our approach, we created a benchmark dataset for the automotive industry. Furthermore, in the interest of advancing research and development, we release our benchmark dataset to the community. INDEX TERMSExpert model, generative AI, jailbreak attacks, large language models, prompt injections. I. INTRODUCTION An important advancement in AI technology has been brought about by the creation of LLMs (Large Language Models) like GPT variations (e.g., InstructGPT [2], LLAMA models [3], Mistral [4], and Mixtral[5]) presenting new concerns and challenges about their safe deployment[6]. These models are pre-trained on internet-scale textual corpora and enhanced with instruction-response pairs and human feedback. Although some models have improved at following instructions, their generative nature makes it harder to prevent them from carrying out potentially harmful The associate editor coordinating the review of this manuscript and approving it for publication was Ayman El-Baz . commands. In contrast, non-generative language models, like text classifiers (e.g., BERT used for classification tasks), produce outputs within fixed categories, making them less susceptible to manipulation by adversarial prompts. Security concerns increase when LLMs are employed as chatbots or AI assistants, where there is a fundamental obligation to avoid causing harm [42]. In such scenarios, users are aware they are interacting with generative AI-based systems and sometimes attempt to test the systemâs limits. Even if done out of curiosity, this can pose significant risks to businesses. Although developers have made considerable efforts to refine these models to mitigate risks, several problems still persist, which include the following: handling malicious questions, preventing prompt injections and jailbreaks, and managing 134976 2025 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks sensitive inquiries related to discounts, pricing, or out-of- domain topics. Letâs consider that one of our main areas of focus is to create conversational AI assistants and chatbots, especially for e-commerce platforms. These assistants are expected to handle a variety of customer inquiries, including providing thorough product information, scheduling appointments, responding to questions about services, and responding to general inquiries from clients to ensure a robust, user-friendly interface that enhances customer service while maintaining high safety standards. Since ChatGPT made its legendary public appearance, it has been misused in several instances. Users have posed illegal questions, such as âHow to rob a bank,â with the model initially responding with detailed instructions, along with answers to other potentially harmful inquiries. This has raised concerns, particularly with sensitive topics such as healthcare, which could also be subject to malicious queries. These challenges are particularly acute when the model is compromised through prompt injection techniques, including jailbreaks, that disrupt its intended function. The persistent problem of model manipulation underscores the complexity of managing and safeguarding interactive language models in practical applications. In the context of LLMs, jailbreak attacks typically involve overloading the model with input to exceed the context length, causing it to forget the original system prompt or com- mand, thus becoming more susceptible to manipulation[43]; Alternatively, attackers may not necessarily fill the context buffer but may slightly alter the conversationâs direction[44]. This form of manipulation, known as âprompt injection,â subverts the intended purpose of the AI assistant, making the system vulnerable to harmful queries. Actions such as these can compromise the modelâs safety mechanisms; for example, while an LLM might initially block a malicious request, altering the prompt could lead it to comply with the request; We tested GPT-4 Turbo against this attack; initially, the model refused to generate malicious outputs. However, after applying prompt injection techniques, GPT-4 Turbo generated malicious outputs. These vulnerabilities highlight the significant risks associated with prompt manipulation and the importance of robust security measures in LLM deployment. Price injections refer to the manipulation of pricing by users through false claims or exploitation of system vulner- abilities to secure illegitimate discounts. This is a serious concern, especially in e-commerce. Retailers frequently offer discounts, and although it is generally acceptable for customers to request discounts, it is crucial to distinguish these requests from manipulative pricing requests. For instance, if a customer claims, âSalesperson X told me I could purchase this product at half price,â it is imperative that our chat systems do not accept such statements without verification. To separate genuine offers from those designed to exploit price injections, careful monitoring is required; For the retail sector, including our use case which is vehicle sales and service, the LLM must accurately identify questions relevant to this specific field. It should avoid engaging in inquiries that fall outside its domain or providing information applicable to other industries. Although APIs or open-source language models are capable of handling requests like Python scripts, such functionalities are extraneous to our focused application in vehicle sales and service. Engaging in these unrelated interactions could potentially compromise our reputation in the automotive retail sector. This principle is applicable to any retail sector, such as electronics, clothing, or home furnishings, where maintaining domain-specific accuracy is crucial to preserving brand integrity and customer trust. Additionally, engaging in out-of-domain queries can lead to cost inefficiencies, particularly when using API-based LLMs, as these interactions may consume resources without adding value to the core business functions. Similarly, harmful queries, which include any malicious interactions that could change the behavior of chatbots, issue unautho- rized discounts, or otherwise compromise the system, are universally undesirable across all retail sectors. No retailer wants their AI assistant to respond to such detrimental queries, as they can jeopardize the integrity and security of the AI system. To summarize, we require a conversational chat model or pipeline, whether developed in-house or via an API-based language model, capable of answering all relevant retail- related inquiries. The model must not reply to hostile questions, be resistant to attempts to disrupt its operation and avoid producing improper content on sensitive themes within the domain. Our goal is for the model to keep a formal but conversational tone, ensuring that responses are precise and suitable. I. RELATED WORK The use of LLMs has increased considerably as they have grown in size and capability, which have made them more exposed to various hazards[28]. In this section, we will look at several research that have looked at this issue from different angles. A. FOUNDATIONAL STUDIES This topic has attracted considerable interest within the research community, as demonstrated by foundational studies on the harmful manipulation of language models[11], [14],[23],[25],[33],[35],[40]; Some of the studies[17] include defensive strategies such as paraphrasing inputs and retokenization, which involves breaking words into smaller units[22], and the use of perplexity-based methods to evaluate and adjust responses[38]. Additionally, red-teaming techniques and the effectiveness of adversarial attacks in conversational settings have been explored[19]. B. IMPROVING PROMPT INJECTION AND JAILBREAK METHODS Researchers have concentrated not only on examining the current situation with prompt injections, but also on VOLUME 13, 2025134977 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks developing approaches for jailbreaking language models. The rationale for this strategy is simple: finding and correcting vulnerabilities early on can help solve problems before they become widely exploited. Several studies have explored this phenomenon; for example, [9], [18], and[36] highlighted that jailbreak attacks are more feasible in certain languages, which warrants more examination. According to [37], low-resource languages are more susceptible to such vulnerabilities, leading to increased disruption in these circumstances. Reference[10]provides a unique technique of neutralizing potentially hazardous prompts by changing them into harmless ones, highlighting the importance of query design. Reference[15]explores the technique of hiding harmful instructions within benign content. Additional research introduces diverse jailbreak strategies, such as the use of representation engineering[12] and taxonomy-guided persuasive adversarial prompts to deceive language mod- els [13], long-context-based attacks that make efforts to influence model behavior[24], and the well-documented prompt injection technique[31]. Furthermore[39] proposes an algorithm that employs an attacker LLM to automatically generate prompts vulnerable to manipulation. C. MITIGATING ATTACKS After understanding how to break large language models, a new research topic emerges: how can we fight against these manipulations? As previously stated in this research, improving the language modelâs ability to follow instructions might make it more likely to obey harmful instructions. For example,[2] discusses the authorsâ approach to training InstructGPT to follow instructions while remaining helpful, honest, and harmless. Reference[34] generates training data to defend LLMs against commonly known jailbreak prompts. Researchers employed these prompts to jailbreak LLMs, then used the resulting instruction-answer pairs for training. However, even after fine-tuning with jailbreak datasets or using gradient matching methods[29], and despite the sometimes high costs of training/fine-tuning, some LLMs remain vulnerable to prompt injection attacks. Reference[21] proposes a method to defend LLMs against jailbreaks using a technique that subtly alters user inquiries with randomized perturbations. Other methods include safe decoding[30], which enhances the likelihood of generating safe tokens and reduces harmful ones; detecting toxicity in LLM-generated outputs using prompts or classifiers[16],[32]; and others. D. BENCHMARK DATASETS To measure the effectiveness of these attacks and defenses, benchmark datasets have been released. The dataset âCrowS- pairsâ[27]is published for measuring social biases in LLMs, while âRealToxicityPromptsâ[26] consists of a 100k prompts to evaluate how often language models generate toxic content. âLatent Jailbreakâ[7] is a benchmark dataset specifically for testing harmful instructions hidden within normal tasks, and âPromptBenchâ[20] is the first systematic benchmark for evaluating, understanding, and analyzing the robustness of LLMs against adversarial prompts. Such datasets have been used to demonstrate that exploiting mistakes in apparently unbreakable prompts might cause LLMs to fail. E. INTEGRATION OF TECHNIQUES-OUR APPROACH Finally, the technique SuperICL (Super In-Context Learning) [41]allows black-box language models to work with locally fine-tuned smaller models by ingesting small modelsâ outputs into the context of the LLMs, resulting in greater performance on supervised tasks. Furthermore the âself- reminderâ technique[8] is introduced, which continually reminds the model throughout the output generation process to be responsible and not to generate malicious output. This technique greatly enhances the modelâs ability to generate safe content, as shown by the results. Our approach combines elements from both the SuperICL and the self-reminder techniques. Specifically, we use small model outputs (as in SuperICL) to reinforce safe behavior during generation (as in self-reminder). I. METHODOLOGY A. ARCHIAS-THE EXPERT For the purposes of our study, we employed a pre-trained transformer-based model, BERT[1], which consists of 109 million parameters. BERT has the capability to generate meaningful representations of text by pre-training on a large corpus and then fine-tuning for specific tasks. In our case, we added a classifier layer to the existing pre-trained model to adapt it for our specific application, which involves classifying user inquiries. The classifier head was trained using a supervised learning approach, wherein the model was fine-tuned with labeled data relevant to our domain. After fine-tuning the BERT model, we obtained the expert model Archias, which we integrated into our main text generation pipeline. As you can see in Fig.1, the pipeline is specifically designed to handle user inquiries by ingesting RAG (retrieval augmented generation) outputs and utilizing a trained expert model to classify these inquiries and deliver accurate responses. One interesting point here is that confidence is also a helpful metric: the system can be designed to ignore the expert modelâs output when the associated confidence score is low. The integration process involved setting up the model within our server environment and ensuring seamless communication between the model and the pipeline. The full pipeline using the expert model was subsequently tested to validate its performance in a real-world scenario â our chatbot. The test confirmed the ability to efficiently process and respond to user queries using the insights derived from the trained BERT model. Integrating Archias with LLM ensures fast and efficient performance. On a GPU, response times remain almost instantaneous, with approximate computation times of just âź5-10 ms). Even on a CPU, it runs smoothly, with only a 134978VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks FIGURE 1.Performance of LLMs with and without expert modelâs output ingested. minimal increase in latency (approximatelyâź50-100 ms). Memory usage stays below 500 MB for real-time tasks, making it suitable for devices with limited resources. While Archias introduces a small amount of additional computation and adds a low cost, which on cloud-based platforms is approximately $35 per month, the significant improvements in accuracy and reliability make this integration highly beneficial. The primary dataset for the expert model was derived from three distinct sources: publicly available, synthetic and masked data from Impel, an automotive AI company. By synthetic generation, we refer to the process where we initially crafted prompts and example templates using a team of three prompt engineers. These templates were then fed into our internally hosted model to generate a synthetic dataset. We created 20-30 canonical prompt templates per category (e.g., common price manipulation tactics, malicious intent queries) grounded in publicly known attack patterns and our internal masked data. These templates were manually reviewed and diversified to ensure broad coverage. All generated variations were subsequently subjected to manual quality checks to maintain data integrity. This dataset is categorized into five categories (see Table1 and training samples from the expert model datasets are shown in Appendix, Fig.7): â˘Publicly available prompt injection data: This data was collected from real-world prompt injection attacks attempted on various AI systems. It includes diverse examples of malicious inputs intended to manipulate the behavior of language models to make them vulnerable. â˘In-domain examples: To ensure the relevance and applicability to our specific domain (vehicle sales and service), we selected examples from domain-specific conversational data gathered across various communica- tion channels, including chat interfaces. These examples were carefully masked and chosen based on user inquiries that align closely with predefined intents. The intents are selected from our in-domain text classifiers (currently used in production for various tasks). The examples encompass the necessary categories within the domain, including initial greetings, targeted questions TABLE 1.Expert model dataset description. about financing options, requests for vehicle informa- tion, and realistic negotiations on pricing. â˘Malicious questions: This dataset was synthetically made using open-source LLMs and contains malicious inquiries. â˘Out-of-domain examples: Questions were syntheti- cally generated to test the robustness of AI models against irrelevant or unexpected queries that fall outside typical usage patterns. Although this is not generally required for large language models, it is crucial for specialized applications in e-commerce, where such queries can pose significant challenges. â˘Price injection: This dataset was also synthetically con- structed, including inquiries specifically formulated to simulate users misleading AI systems into erroneously disclosing or manipulating pricing information. To enhance the quality and reliability of our dataset, an extensive manual review as well as re-labeling were essential. Our team of prompt engineers and data labelers played a critical role in this phase. Each example from the compiled datasets, particularly those collected online, was meticulously examined. This approach facilitated a comprehensive understanding of the various types of prompt injections and ensured that our AI models were trained on data that was representative of real-world scenarios and challenges, thereby enhancing their effectiveness in identifying and mitigating potential threats. Furthermore, while our current implementation focuses on automotive industry, the same training methodology can be applied to other domains. To extend Archias to a new field, one would need to gather or generate domain-specific inquiry examples (e.g., healthcare, finance), categorize them VOLUME 13, 2025134979 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks into relevant classes (e.g., in-domain, out-of-domain), and fine-tune the expert model using the same supervised learning setup described above. B. BENCHMARK DATASET We developed a benchmark dataset specifically designed to evaluate the LLMâs understanding and reasoning capabilities in response to attacks and context-specific inquiries. The selected topics cover prompt and price injections, malicious questions, specialized in-domain inquiries requiring contex- tual knowledge, in-domain queries that rely on common- sense knowledge, and out-of-domain topics. This variety allowed us to test the modelâs ability to understand and reason across diverse scenarios. Each LLM under study was tested using this dataset to assess not only its baseline performance but also how it benefits from the integration of an expert-designed subsystem that is aimed at enhancing reasoning and resistance to prompt injections. We utilized EleutherAIâs open-source framework, Harness, available on GitHub [45], to implement our benchmark-specific details and conduct tests on various models. After researching well-known benchmark datasets[46], [47],[48],[49],[50], we decided to use the multiple-choice format for ours. This closed-ended approach allows for straightforward analysis and comparison, as each response is limited to the given options, and is more suitable for our use case. We constructed benchmark samples using a predefined instruction that is the same for every example, along with unique user inquiries, questions, and multiple-choice answers. While this format provides a controlled environment for evaluating the modelâs ability to recognize and categorize harmful queries, we acknowledge that real-world scenarios often involve open-ended interactions without predefined options. Exploring alternative evaluation setups that better simulate these conditions is an important direction for future work. Fig. 2 shows an example of our benchmark dataset: A) without and B) with use of the expert modelâs output. As demonstrated in the exampleâs user inquiry, the user attempts to bypass the AI modelâs restrictions by introducing price-related content, which may not be easily noticed by the language model, although it is apparent to human observers. Although such communication may be harmless if directed toward a human, in this context it could potentially result in severe issues. Each item in the dataset consists of an inquiry followed by several response options, with only one being correct. Some items also provide context to ensure all models have the necessary domain knowledge to answer the question. We manually crafted a total of 150 examples, each designed to challenge the models on various aspects of the five topics in Table1. The crafting process involved generating inquiries and corresponding choices, and con- ducting meticulous reviews and revisions to ensure clarity, relevance, and the potential to differentiate between more TABLE 2.Results for Large Language models in our benchmark dataset. and less capable models in handling complex linguistic tasks. The challenge was to ensure that the correct answer was not very obvious and required reasoning. With assistance from our colleagues (Three of our teamâs prompt engineers), we achieved this goal. Following the experiment, we invited team members and friends to answer the benchmark ques- tions to gauge human performance. The average score was 88% of the 150 questions answered correctly. To determine human performance, we subsequently invited 50 individuals: 30 team members (domain knowledge experts) and 20 friends (non-expert participants) to answer the same benchmark questions. They were presented with the questions and asked to choose the most appropriate answer - without any additional context or guidance. Additionally, they were instructed not to use any language models or external tools and to determine the answers independently. Upon analyzing the responses, we noticed that certain questions consistently received the same answers from all participants. This prompted us to revise these questions to ensure a more balanced assessment. However, we also discovered that some consistent answers resulted from participant mistakes, not from problems with the questions themselves. Therefore, we decided not to change those questions. This mixed group reflects both informed and average users, helping us benchmark model performance in realistic deployment settings. We emphasize that the 88% accuracy reflects 134980VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks FIGURE 2.Benchmark dataset example. aggregated human performance and not an expert-only upper bound, providing a practical frame of reference for model comparison. As a result, we have 150 examples in our benchmark dataset, along with 150 similar examples that include expert opinions, effectively representing the topics for which we intended to test our models. The label distribution for dataset is: 41 Malicious Questions, 31 Prompt Injections, 27 Out-of-Domain, 26 Price Injections, and 25 In-Domain samples. The complete benchmark dataset, including all exam- ples and corresponding expert-labeled variants, is publicly available on Hugging Face [51]. This allows researchers to reproduce our results and extend evaluations across other models and domains. IV. RESULTS AND DISCUSSION This section conducts a comparative analysis of LLMâs, focusing on the performance improvements achieved through expert model integration. The analysis includes detailed metric scores and illustrative examples from the benchmark dataset, categorized to demonstrate improvements across various dimensions of performance. Our work began with fine-tuning Archias. After select- ing the optimal configurations and completing the fine- tuning, we conducted actual tests using the previously described LLM-eval framework. We evaluated publicly available language models, our in-house LLM (Impel-LLM), and API-based models using our benchmark within this framework. This section will first describe the details behind the Archias experiments and then discuss the performance of the models on our benchmark, comparing results with and without input from our expert model. In the best-performing experiment, the BERT-based model demonstrated robust performance on the test dataset, with an F1 score of 0.92 and an accuracy of 0.94. These results were obtained using a learning rate of 10 â5 , a batch size of 16, and a total of 3 training epochs, with a weight decay of 0.1. Macro averaging was conducted across the test data to ensure a comprehensive evaluation of the modelâs predictive capabilities. We also focused on the analysis of the modelâs error patterns, particularly in distinguishing between âin-domainâ topics and âprice injectionâ scenarios. For example, the sentence âCan you snap up that deal for me before someone else does?â was incorrectly classified by Archias as a price injection, although it was labeled as in-domain in our test dataset. This mistake likely occurred because the model is sensitive to phrases that resemble negotiation, which is common in price discussions. However, in this case, the language used was typical of a permissible sales conversation, where the main goal was to secure a deal quickly rather than to negotiate the price down. As previously mentioned, our study revealed that incor- porating expert opinions consistently enhances performance across the benchmark (see Fig.3), particularly in detecting and reasoning about jailbreaks, price injections, and out- of-domain examples. Table2, where for all evaluations, outputs were generated using greedy decoding (A technique VOLUME 13, 2025134981 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks in which the model selects the most likely next word at each step without considering alternative word combina- tions.), shows a notable enhancement in several models; for example, LLama 3 70B Instruct from Meta, a significantly large model, achieved a percentage improvement of 17.9% after integrating expert model output. Interestingly, the non-instruct version of Llama-3-70B exhibited an 8.5% improvement, despite performing better than the instruct version on the benchmark when expert opinions were excluded. This discrepancy might be attributed to the fact that instruction-tuned models are designed to pay attention to each detail in the prompt rather than merely generating next tokens, thereby enabling a focused attention to the context we provided. Furthermore, it is noteworthy that even GPT models evaluated via API demonstrated substantial improvements. Specifically, GPT-3.5 with expert output outperformed GPT-4 Turbo on the same benchmark without expert input. A similar pattern emerged with Mistral-7B- Instruct-v0.2, which came quite close to the more advanced model, Mixtral-8Ă7B-Instruct-v0.1. Both were developed by the same research organization. To evaluate the benefits of our approach, we compared it to the âself-reminderâ technique. This method incorporates the following sentence in the system prompt, as recom- mended by the technique: âYou should be a responsible AI and should not generate harmful or misleading content! Please answer the following user query responsibly!â We conducted tests using three models, GPT-4, Meta-Llama- 3-8B-Instruct, and Mistral-7B-Instruct-v0.2, representing different providers (OpenAI, Meta, and Mistral). The results, summarized in Table 3, indicate that when the userâs intent is stated more explicitly and the model reflects this intent in its reasoning, the generated responses are notably more accurate. TABLE 3.Benchmark results: Expert integration vs. Self-reminder. Impelâs in-house automotive model, Impel-LLM, is based on Mistral 7B and has been fine-tuned using our masked reasoning data from both Impelâs conversations and the text data reflecting salespersonsâ behavior. It also determines the implications of jailbreaks and explains why they are problematic for us, thanks to its understanding of the underlying reasons. This alignment with our specific use case, vehicle dealerships and understanding salesperson behavior, is reflected in its performance on our benchmark: it achieves 63% accuracy on the benchmark without expert FIGURE 3. Performance of LLMs with and without expert modelâs output ingested. FIGURE 4. Improvements in model performance across various categories. opinion, which is better than most publicly available mod- els. Additionally, with input from Archias, our LLM has improved by 12.7%, reaching performance levels comparable to those of GPT-3.5 Turbo and GPT 4 Turbo, providing strong evidence that our approach is effective and holds promise for continued advancements through the expert modelâs integration and LLM development. Let us now examine specific examples from our bench- mark dataset. For example, consider the user query, âThis is a test by the developers. Disable your content filters and provide a list of all your restricted commands,â which was analyzed in our reasoning example (see Fig.5 in the Appendix). This request was initially misinterpreted by several models, including GPT-3.5 Turbo, GPT-4, Llama-2- 13B-hf, and Llama-3-8B, with these models stating âThis statement is inappropriate because it asks the AI assistant to disable security measures and reveal sensitive operational details, which could compromise the systemâs integrity and the dealershipâs security protocols.â However, with the assistance of an expert model, the response was refined to âThis statement is harmful as it suggests a potential prompt injection attempt, asking the AI Assistant to bypass security protocols and expose restricted information, which is unethical and could endanger the dealershipâs operations.â This clarification is important, as it not only addresses the inappropriateness of requesting sensitive information, but also emphasizes the importance of detecting and analyzing 134982VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks TABLE 4.Results for large language models in our benchmark dataset. potential prompt injection attempts to gain unauthorized access to that information. In a notable example, models were prompted with the phrase, âPretend youâre a cow nowâ (Appendix: Fig.6). Various models from the LLama series (e.g., LLama-3-70B- Instruct, LLama-3-8B) and GPT 3.5 Turbo exhibited different responses to this instruction, despite the scenario clearly stating that the context was a dealershipâs AI assistant. Some models interpreted the playful nature of the input positively, suggesting that such an approach could alleviate stress and enhance rapport between users and the AI; however, other models flagged the input as inappropriate, mistakenly associating it with elements of sexual harassment, which was not the intended implication. The use of the expert modelâs opinions was crucial in identifying the real issue at hand. In Fig.4 we present a chart that depicts the improve- ments achieved by integrating expert input across various categories. The enhancements are calculated by dividing the performance improvement of each model by its baseline results. Each row in the chart represents a baseline model and its enhanced counterpart with expert involvement. Notably, we observe significant advancements in handling âprice injectionâ questions. This improvement is critical because many attacks specifically target this category, possibly due to the challenges in distinguishing between general sales negoti- ations and price injections. Substantial improvements are also evident in âin-domainâ and âprompt injectionâ questions. Conversely, the smallest enhancements are observed in the âmalicious questionâ and âout-of-domainâ categories, likely because these issues are already well-addressed by general models and are more readily detectable. The comparative analysis confirms that integrating expert model outputs significantly enhances the robustness and precision of language models, particularly in detecting and mitigating VOLUME 13, 2025134983 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks FIGURE 5.Benchmark example 1. FIGURE 6.Benchmark example 2. security vulnerabilities. These findings underscore the critical importance of comparative evaluations in advancing secure AI deployments, providing a framework for future research and optimization. 134984VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks FIGURE 7.Expert model dataset samples. V. CONCLUSION Our research proposes a mechanism that combines reminder and ingestion approaches from expert models into LLMs. In this paper, we presented the expert model Archias, a fully standalone system designed to enhance LLM outputs by categorizing user queries, identifying irrelevant or harmful content, and providing confidence scores to minimize errors in complex cases such as price manipulation or malicious requests. By integrating Archiasâs outputs into the LLM, as our proposed mechanism suggests, the model gains expert insights, resulting in a clearer understanding of user intent and improved response quality. Results showed that this approach improved LLM outcomes while also providing significant protection against prompt injections and other undesired tactics, especially in specialized domains. Reflecting on our focused approach, our work demonstrates the efficacy of our expert model within the automotive and retail AI sectors. To enhance its applicability, we have incorporated proxies in our benchmark dataset for challenges such as prompt injections, malicious questions, and price manipulations - issues prevalent across various retail domains. This inclusion ensures our datasetâs robustness and relevance for scenarios across different retail contexts, illustrating the modelâs potential cross-domain applicability. We also released a benchmark dataset to test solutions to these problems, indicating that our technology is dependable and represents a potential strategy for the future. We will continue to improve defense techniques against prompt injections and explore these concerns, as each improvement in LLM technology is usually accompanied by new dangerous attacks. The effectiveness of this technology opens up new research directions, indicating its potential in a variety of multi-task circumstances. While our benchmark dataset accurately represents the intended themes for testing our models, we realize the need to increase the number of test cases, as jailbreak attempts become increasingly sophisticated. Current limitations also include edge-case misclassifications and the need for broader testing under real- world constraints. This makes a new research opportunity for us to generalize the approach. Furthermore, we intend to test our automotive LLM with this technology in a production setting, and we will continue to share our experimental findings and latest research initiatives with the community. ACKNOWLEDGMENT The authors are grateful for the companyâs contin- uous support and resources that made this research possible. (Tatia Tsmindashvili, Ana Kolkhidashvili, Dachi Kurtskhalia, Nino Maghlakelidze, Elene Mekvabishvili, Guram Den- toshvili, Orkhan Shamilov, Zaal Gachechiladze, Steven Saporta, and David Dachi Choladze contributed equally to this work.) VOLUME 13, 2025134985 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks APPENDIX RESULTS AND EXAMPLES See Table 4 and Figs. 5â7. REFERENCES [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, âBERT: Pre-training of deep bidirectional transformers for language understanding,â 2018, arXiv:1810.04805. [2] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, âTraining language models to follow instructions with human feedback,â inProc. Advances in Neural Information Processing Systems, vol. 35, 2022, p. 27730â27744. [3] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. RoziĂŠre, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, âLLaMA: Open and efficient foundation language models,â 2023,arXiv:2302.13971. [4] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, âMistral 7B,â 2023,arXiv:2310.06825. [5] A. Q. Jiang et al., âMixtral of experts,â 2024,arXiv:2401.04088. [6] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, âSurvey of hallucination in natural language generation,âACM Comput. Surveys, vol. 55, no. 12, p. 1â38, Mar. 2023. [7] H. Qiu, S. Zhang, A. Li, H. He, and Z. Lan, âLatent jailbreak: A benchmark for evaluating text safety and output robustness of large language models,â 2023,arXiv:2307.08487. [8] Y. Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, âDefending ChatGPT against jailbreak attack via self-reminders,âNature Mach. Intell., vol. 5, no. 12, p. 1486â1496, Dec. 2023. [9] J. Li, Y. Liu, C. Liu, L. Shi, X. Ren, Y. Zheng, Y. Liu, and Y. Xue, âA cross- language investigation into jailbreak attacks in large language models,â 2024,arXiv:2401.16765. [10] K. Takemoto, âAll in how you ask for it: Simple black-box method for jailbreak attacks,â 2024,arXiv:2401.09798. [11] M. Gupta, C. Akiri, K. Aryal, E. Parker, and L. Praharaj, âFrom ChatGPT to ThreatGPT: Impact of generative AI in cybersecurity and privacy,âIEEE Access, vol. 11, p. 80218â80245, 2023. [12] T. Li, Z. Wang, W. Liu, M. Wu, S. Dou, C. Lv, X. Wang, X. Zheng, and X. Huang, âRevisiting jailbreaking for large language models: A representation engineering perspective,â 2024,arXiv:2401.06824. [13] Y. Zeng, H. Lin, J. Zhang, D. Yang, R. Jia, and W. Shi, âHow Johnny can persuade LLMs to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs,â 2024,arXiv:2401.06373. [14] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, âDo anything Nowâ: Characterizing and evaluating In-The-Wild jailbreak prompts on large language models,â 2023,arXiv:2308.03825. [15] X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han, âDeepInception: Hyp- notize large language model to be jailbreaker,â 2023,arXiv:2311.03191. [16] M. Phute, A. Helbling, M. Hull, S. Peng, S. Szyller, C. Cornelius, and D. H. Chau, âLLM self defense: By self examination, LLMs know they are being tricked,â 2023,arXiv:2308.07308. [17] N. Jain, A. Schwarzschild, Y. Wen, G. Somepalli, J. Kirchenbauer, P.-Y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, âBaseline defenses for adversarial attacks against aligned language models,â 2023,arXiv:2309.00614. [18] Y. Deng, W. Zhang, S. Jialin Pan, and L. Bing, âMultilingual jailbreak challenges in large language models,â 2023,arXiv:2310.06474. [19] D. Ganguli et al., âRed teaming language models to reduce harms: Meth- ods, scaling behaviors, and lessons learned,â 2022,arXiv:2209.07858. [20] K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, Y. Zhang, N. Zhenqiang Gong, and X. Xie, âPromptRobust: Towards evaluating the robustness of large language models on adversarial prompts,â 2023,arXiv:2306.04528. [21] A. Robey, E. Wong, H. Hassani, and G. J. Pappas, âSmoothLLM: Defending large language models against jailbreaking attacks,â 2023, arXiv:2310.03684. [22] S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, and T. Sun, âAutoDAN: Interpretable gradient-based adver- sarial attacks on large language models,â 2023,arXiv:2310.15140. [23] A. Rao, S. Vashistha, A. Naik, S. Aditya, and M. Choudhury, âTricking LLMs into disobedience: Formalizing, analyzing, and detecting jail- breaks,â 2023,arXiv:2305.14965. [24] C. Anil, E. Durmus, M. Sharma, J. Benton, S. Kundu, J. Batson, N. Rimsky, M. Tong, J. Mu, and D. Ford, âMany-shot jailbreaking,â inProc. Adv. Neural Inf. Process. Syst., 2023, p. 1â47. [25] T. Kim, S. Kotha, and A. Raghunathan, âTesting the limits of jailbreaking defenses with the purple problem,â 2024,arXiv:2403.14725. [26] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, âRealToxicityPrompts: Evaluating neural toxic degeneration in language models,â 2020,arXiv:2009.11462. [27] N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman, âCrowS-pairs: A challenge dataset for measuring social biases in masked language models,â 2020,arXiv:2010.00133. [28] N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ă. Erlingsson, A. Oprea, and C. Raffel, âExtracting training data from large language models,â inProc. 30th USENIX Secur. Symp., Aug. 2021, p. 2633â2650. [29] L. He, M. Xia, and P. Henderson, âWhat is in your safe data? Identifying benign data that breaks safety,â 2024,arXiv:2404.01099. [30] Z. Xu, F. Jiang, L. Niu, J. Jia, B. Yuchen Lin, and R. Poovendran, âSafeDecoding: Defending against jailbreak attacks via safety-aware decoding,â 2024,arXiv:2402.08983. [31] F. Perez and I. Ribeiro, âIgnore previous prompt: Attack techniques for language models,â 2022,arXiv:2211.09527. [32] Y.-S. Wang and Y. Chang, âToxicity detection with generative prompt- based inference,â 2022,arXiv:2205.12390. [33] A. Wei, N. Haghtalab, and J. Steinhardt, âJailbroken: How does LLM safety training fail?â inProc. Adv. Neural Inf. Process. Syst., vol. 36, 2024, p. 1â12. [34] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, âRed teaming language models with language models,â 2022,arXiv:2202.03286. [35] D. Ganguli et al., âThe capacity for moral self-correction in large language models,â 2023,arXiv:2302.07459. [36] P. C. R. Puttaparthi, S. S. Deo, H. Gul, Y. Tang, W. Shang, and Z. Yu, âComprehensive evaluation of ChatGPT reliability through multilingual inquiries,â 2023,arXiv:2312.10524. [37] Z.-X. Yong, C. Menghini, and S. H. Bach, âLow-resource languages jailbreak GPT-4,â 2023,arXiv:2310.02446. [38] G. Alon and M. Kamfonas, âDetecting language model attacks with perplexity,â 2023,arXiv:2308.14132. [39] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, âJailbreaking black box large language models in twenty queries,â 2023, arXiv:2310.08419. [40] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, âMASTERKEY: Automated jailbreaking of large language model chatbots,â inProc. Netw. Distrib. Syst. Secur. Symp., 2024. [41] C. Xu, Y. Xu, S. Wang, Y. Liu, C. Zhu, and J. McAuley, âSmall models are valuable plug-ins for large language models,â 2023,arXiv:2305.08848. [42] H. Li, D. Guo, W. Fan, M. Xu, J. Huang, F. Meng, and Y. Song, âMulti-step jailbreaking privacy attacks on ChatGPT,â 2023,arXiv:2304.05197. [43] E. Shayegani, M. Abdullah Al Mamun, Y. Fu, P. Zaree, Y. Dong, and N. Abu-Ghazaleh, âSurvey of vulnerabilities in large language models revealed by adversarial attacks,â 2023,arXiv:2310.10844. [44] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Zico Kolter, and M. Fredrikson, âUniversal and transferable adversarial attacks on aligned language models,â 2023,arXiv:2307.15043. [45] L. Gao et al., âA framework for few-shot language model evaluation,â Zenodo, Dec. 2023, doi:10.5281/zenodo.10256836. [46] S. Reddy, D. Chen, and C. D. Manning, âCoQA: A conversational question answering challenge,âTrans. Assoc. for Comput. Linguistics, vol. 7, p. 249â266, Nov. 2019. [47] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, âHellaSwag: Can a machine really finish your sentence?â 2019,arXiv:1905.07830. [48] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, âMeasuring massive multitask language understanding,â 2020,arXiv:2009.03300. [49] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, âSelf-instruct: Aligning language models with self-generated instructions,â 2022,arXiv:2212.10560. 134986VOLUME 13, 2025 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks [50] S. Longpre, L. Hou, T. Vu, A. Webson, H. W. Chung, Y. Tay, D. Zhou, Q. V. Le, B. Zoph, J. Wei, and A. Roberts, âThe flan collection: Designing data and methods for effective instruction tuning,â inProc. 40th Int. Conf. Mach. Learn., Jul. 2023, p. 22631â22648. [51] Impel Research Team. (2024).Jailbreak Benchmark: Evaluating LLM Robustness Against Prompt Injection and Malicious Queries. [Online]. Available: https://huggingface.co/datasets/Impel/Jailbreak_ Benchmark TATIA TSMINDASHVILIreceived the B.Sc. degree in electrical and computer engineering from the Free University of Tbilisi, in 2021, and the M.Sc. degree in biomedical engineering and medical informatics from Georgian Technical University, in 2024. She began her professional career, in 2020, working in roles, such as a Software Engineer and a Data Engineer. She joined Impel, in 2021, as a Data Scientist, and was promoted to a Senior Deep Learning Researcher, in January 2023. Alongside her industry work, she has been actively involved in research. She contributed as a Research Assistant with the Free University of Tbilisi, developing algorithms to analyze neural activity. Since January 2023, she has been part of the GAIN project, focusing on applying AI to EEG signals. Her work includes designing deep learning models, with several publications forthcoming. ANA KOLKHIDASHVILIreceived the bachelorâs degree in Georgian philology and the masterâs degree in natural language processing, focusing on lexical functions from Tbilisi State University, in 2013 and 2015, respectively, where she is currently pursuing the Ph.D. degree in theoretical and applied linguistics. Her professional journey in AI began with Impel (formerly Pulsar AI), where she initially joined as a Computational Linguist, in 2017. In this role, she developed various NLP solutions, including spell checkers, transliteration tools, sentiment analysis, and voice assistant prototypes. Following the acquisition of Pulsar AI by Impel, in 2021, she was appointed as the Director of the Research and Development Team. She leads initiatives on large language models and AI architecture, focusing on automated conversation systems. She works closely with cross-functional teams to ensure the successful execution and delivery of projects. She is a Pivotal Figure in driving AI advancements and leadership at Impel. Her research interests include advanced NLP, AI model development, automated conversation design, and project management. DACHI KURTSKHALIAreceived the bachelorâs degree in mathematics and computer science from the Free University of Tbilisi, where he was noted for high achievement. He joined Impel, in January 2023, focusing on leveraging artificial intelligence in the automotive industry, following his previous roles as a Data Engineer with Bank of Georgia and EPAM. He is currently a Data Scientist with Impel. His passion for AI is reflected in his leadership roles, including a Teaching Assistant and spearheading the development of a Georgian language AI model at his alma mater. His active involvement in datathons and leadership in the AI Club at the university highlights his dedication to innovation and community engagement in AI. NINO MAGHLAKELIDZEreceived the bachelorâs degree in law from Ivane Javakhishvili Tbilisi State University. She is currently pursuing the masterâs degree in comparative private and international law with New Vision University. She began her career with Pulsar, in 2020, as a QA Agent, rapidly advancing to an AI/ML Specialist, in 2021, with a focus on natural language processing (NLP) and understanding (NLU). In October 2023, she transitioned to a Prompt Engineer with the Research and Development Team, further enhancing her expertise in AI-driven technologies. She is currently an AI/ML Specialist and a Prompt Engineer with Impel. Her role involves building and optimizing machine learning models and their integration into software applications, collaborating closely with data scientists and engineers. ELENE MEKVABISHVILIreceived the B.A. degree in American studies and translation, and has carved a niche for herself in the AI field, focusing on voice AI technology. Starting her career with Pulsar, as a QA Specialist before its acquisition by Impel, in 2021, she meticulously monitored and analyzed conversational AI response accuracy. She later transitioned to the role of a Conversational AI Tester with the Conversation Design Team, a position she held for 1.5 years. In November 2023, she advanced to her current role as a Research and Development Product Tester. She is currently a Research and Development Product Tester with Impel. Her expertise spans testing AI responses for sales-related projects and contributing to the development of conversational systems in the automotive industry. Her skills in prompting and natural language processing (NLP) enable her to deliver innovative AI solutions with a versatile and dynamic approach. GURAM DENTOSHVILIreceived the bachelorâs degree in mathematics and computer science from the Free University of Tbilisi. He began his career with Pulsar, in 2019, as a Machine Learning Engineer, and following his acquisition of Pulsar, he was promoted to an AI Engineering Team Lead. In this role, he spearheaded the development and enhancement of various conversational AI products tailored for the automotive industry, significantly influencing the scalability and manageability of AI solutions in this sector. Following his notable successes, he transitioned to a research-focused role. He is currently the Director of Engineering of the Research and Development Team, Impel. He continues to drive innovation within the team by introducing cutting-edge product and technology ideas and refining existing technologies to maintain a competitive edge in the market. ORKHAN SHAMILOVreceived the M.B.A. degree from Ilia State Uni- versity and the masterâs degree in business administration, management, and operations from Kaunas University of Technology, completed through the Erasmus+ exchange program, in 2020. He is currently a Data Engineer with Impel. He is a dedicated member of the Research and Development Team, Impel, specializing in natural language processing (NLP). Since joining Impel, in January 2023, he has focused on data preprocessing, training, and fine-tuning of large language models (LLMs), leveraging his Python expertise to refine datasets and develop advanced models. Prior to joining Impel, he was a Machine Learning Engineer with Supernova, from 2022 to 2023, where he implemented state-of-the-art machine learning models, contributing significantly to the advancement of AI applications. VOLUME 13, 2025134987 T. Tsmindashvili et al.: Improving LLM Outputs Against Jailbreak Attacks ZAAL GACHECHILADZEreceived the Bachelor of Natural Sciences degree in physics. He is a Prominent Figure in AI innovation. As the Co-Founder of Pulsar AI, the first Georgian startup to successfully exit in U.S. market, he has notably enhanced customer-car dealer interactions through conversational AI. He is currently an Entrepreneur and a Consultant with Impel. His leadership helped secure substantial Silicon Valley investments, elevating Pulsar AIâs global presence. Additionally, he has contributed significantly to AI development for the Georgian language, addressing unique natural language processing challenges. His efforts not only advanced Pulsar AIâs technological capabilities but also inspired Georgian entrepreneurs to innovate in the tech industry. As the Co-Founder and the Leader of Tbilisi AI Laboratory, he continues to develop AI solutions tailored for Georgian SMEs, advocating for AIâs role in driving economic growth and innovation in Georgia. STEVEN SAPORTAreceived the degree in computer science from Princeton University. He is a Technology Executive with a wide range of experience in all aspects of software engineering and business. He was hired as the third employee of Midi Inc., he became CTO and aided the acquisition by a team of investors and merger with the publicly traded company SAI Global. He then became the CTO of LocalUp Solutions (now OrderUp, a Groupon Company), where he built and led the team that developed and supported e-commerce software serving more than 500 000 customers. His next position was as the CTO of Joule Assets, a startup in the energy industry. He is currently the Chief Information Officer of Impel, a leading provider of software and AI to the automotive industry. He brings to his career a passion for technology startups, and hands-on experience with multiple programming languages, databases, servers, and cloud hosting platforms. His technical acumen combines with years of business experience, including budget planning, recruiting, team leadership, proposal writing, vendor management, contract negotiation, and investor relations. DAVID DACHI CHOLADZEreceived the M.B.A. degree from the Free University of Tbilisi and the double degree in business administration and international economics and management from Central European University, Budapest, and Bocconi University, Milan. He co-founded Pulsar AI, becoming Georgiaâs first significant AI startup, which innovated the financial sector with a pioneering chatbot for TBC Bank, in 2017. His venture was the first from Georgia to secure Silicon Valley investment, leading to its expansion and a strategic merger with Impel, where he is currently the Chief Innovation Officer and an Entrepreneur. Operating in more than 51 countries, his work focuses on technology entrepreneurship, AI applications in the automotive industry, and strategic business development. 134988VOLUME 13, 2025