Paper deep dive
Exploring Advanced Methodologies in Security Evaluation for LLMs
Jun Huang, Jiawei Zhang, Qi Wang, Weihong Han, Yanchun Zhang
Models: Claude 2, GPT-4
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%
Last extracted: 3/11/2026, 12:54:14 AM
Summary
This paper provides a comprehensive review of security evaluation methodologies for Large Language Models (LLMs), categorizing evaluation frameworks into black-box and white-box approaches, and defining three primary evaluation processes: data-driven evaluation, attack simulation, and formal verification. It also discusses common evaluation metrics and identifies future research directions, such as the need for automated evaluation platforms and improved security assessment for multimodal models.
Entities (6)
Relation Signals (3)
Black-box Evaluation Framework â isa â Security Evaluation Framework
confidence 95% · We explore the security evaluation frameworks that have been developed and implemented, categorizing these frameworks into two types according to the traditional black-box/white-box evaluation
White-box Evaluation Framework â isa â Security Evaluation Framework
confidence 95% · We explore the security evaluation frameworks that have been developed and implemented, categorizing these frameworks into two types according to the traditional black-box/white-box evaluation
Data-driven Evaluation â isa â Security Evaluation Process
confidence 95% · We abstract Data-driven Evaluation, Attack Simulation, and Formal Verification as three types of security evaluation processes for LLMs
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models (LLMs) represent an advanced evolution of earlier, simpler language models. They boast enhanced abilities to handle complex language patterns and generate coherent text, images, audios, and videos. Furthermore, they can be fine-tuned for specific tasks. This versatility has led to the proliferation and extensive use of numerous commercialized large models. However, the rapid expansion of LLMs has raised security and ethical concerns within the academic community. This emphasizes the need for ongoing research into security evaluation during their development and deployment. Over the past few years, a substantial body of research has been dedicated to the security evaluation of large-scale models. This article an in-depth review of the most recent advancements in this field, providing a comprehensive analysis of commonly used evaluation metrics, advanced evaluation frameworks, and the routine evaluation processes for LLMs. Furthermore, we also discuss the future directions for advancing the security evaluation of LLMs.
Tags
Links
- Source: https://arxiv.org/abs/2402.17970
- Canonical: https://arxiv.org/abs/2402.17970
Trouble viewing inline? Open PDF directly â
Full Text
43,664 characters extracted from source content.
Expand or collapse full text
ExploringAdvancedMethodologiesinSecurity EvaluationforLargeLanguageModels JunHuang 1 ,JiaweiZhang 1 ,QiWang 2 ,WeihongHan 1 ,andYanchunZhang 1 1 PengChengLaboratory,Shenzhen,China huangj02,zhangjw01,hanwh@pcl.ac.cn,yanchun.zhang@vu.edu.au 2 InformationScienceandTechnologyCollege,DalianMaritimeuniversity,Dalian, China qiwang1101@163.com Abstract.LargeLanguageModels(LLMs)representanadvancedevo- lutionofearlier,simplerlanguagemodels.Theyboastenhancedabilities tohandlecomplexlanguagepatternsandgeneratecoherenttext,images, audios,andvideos.Furthermore,theycanbefine-tunedforspecifictasks. Thisversatilityhasledtotheproliferationandextensiveuseofnumer- ouscommercializedlargemodels.However,therapidexpansionofLLMs hasraisedsecurityandethicalconcernswithintheacademiccommunity. Thisemphasizestheneedforongoingresearchintosecurityevaluation duringtheirdevelopmentanddeployment.Overthepastfewyears,a substantialbodyofresearchhasbeendedicatedtothesecurityevalua- tionoflarge-scalemodels.Thisarticleanin-depthreviewofthemost recentadvancementsinthisfield,providingacomprehensiveanalysisof commonlyusedevaluationmetrics,advancedevaluationframeworks,and theroutineevaluationprocessesforLLMs.Furthermore,wealsodiscuss thefuturedirectionsforadvancingthesecurityevaluationofLLMs. Keywords:Largelanguagemodels·Securityevaluation·LLMs. 1Introduction Languagemodels(LMs)arestatisticalmachinesdesignedtocomprehendnat- urallanguage,whichcalculatethelikelihoodofwordsequencesbasedonvast collectionsoftext[5,58]. Largelanguagemodels(LLMs)representasignificantadvancementovertheir simplerpredecessors,showcasingenhancedabilitiestounderstandintricatelin- guisticpatternsandgeneratecoherent,contextuallyappropriatetext[1,45,55]. Inadditiontotext,LLMscanalsoprocesspictures[3],audiorecordings[4], andvideos[2].Moreover,theycanbefine-tunedtoexcelatspecifictaskswith utmostprecision.WiththeimpressivegrowthofLLMsâcapabilities,numerous commerciallyavailablelargemodels,suchasOpenAIâsGPTseries,Anthropicâs Claudeseries,GoogleâsT5andPaLM,MetaâsOPT,andBaiduâsErnie[57],have flourishedandfoundextensiveapplications. * SupportedbyPengChengLaboratory. 2J.Huangetal. Fig.1:Proportionalrepresentationofmodelreleaseswithparametersexceeding 1billion,sourcedfromOpenBMB[34]. ThepotentialapplicationsofLLMsarevastandcontinuetogrowasthe technologymatures.FromFig.1,itâsevidentthatthenumberofmodelswith parametersexceeding1billionhasseenasignificantincreasesince2019.This trendcanbetracedbacktoasequenceofdevelopmentsandkeyevents.The stagewassetin2019,highlightedbyMicrosoftâsonebillioninvestmentinOpe- nAI,whichunderscoredthesubstantialcommercialinterestinAIandboosted resourcesforAIdevelopment[11].Themomentumcontinuedinto2020withthe launchofNVIDIAâsA100TensorCoreGPU[26],providingresearcherswithun- precedentedcomputationalpowerandenablingmorecomplexmodeltraining. Soonafter,thereleaseofOpenAIâsGPT-3modelservedasapowerfuldemon- strationofthepotentialoflargemodels[44],spurringfurtherresearchanddevel- opmentinthefield.Aswetransitionedinto2021,innovativetrainingtechniques likemixedprecisiontraining,modelparallelism,anddataparallelismbecame mainstream,facilitatingthetrainingofevenlargermodels.Concurrently,de- spitethesetechnologicaladvancementsandtheevidentpotentialoflarge-scale models,theirrapidproliferationhasalsoraisedconcernswithintheacademic community,particularlywithregardtothesecurityimplicationsofthesemod- els[52].Theseconcernsunderscorethenecessityofongoingresearchintosecure andethicalpracticesinthedevelopmentanddeploymentofLLMs. Researchershaveconductedliteraturereviewsfocusingonthesecurityrisks andevaluationmethodsforLLMs.Weidingeretal.(2022)inGoogleDeepMind outlinedacomprehensivetaxonomyofethicalandsocialrisksassociatedwith large-scalelanguagemodels,providingausefulframeworktoguideresponsible innovationinthisspace[56].However,itlacksaspecificimplementationplan necessaryforconductingasecurityevaluation.Changetal.(2023)presenteda reviewofresearchworkontheevaluationofLLMs[6].Theydetailedaclassi- ficationandsummarybasedonthreedimensions:"whattoevaluate,""where toevaluate,"and"howtoevaluate."Theysummarizedvarioussuccessfuland ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs3 unsuccessfulcasesofLLMsacrossmultipledomains,28popularbenchmarksfor LLMevaluation,andtwoprimaryevaluationmethods:automaticandmanual. ButtheirevaluationfocusedmoreonbenchmarksforevaluatingLLMs,without comprehensiveanalysisoftheevaluationsteps.Huangetal.(2023)haveprovided areviewofthevariousknownvulnerabilitiesofLLMs,consideringbothtechnical methodsandethicalnormstopromotethesafetyandcredibilityofLLMs[18]. TheirworkservesasareferenceforthefurtherdevelopmentofLLMs.Still,their discussionofknownevaluationframeworksisnotcomprehensive.Glukhovet al.(2023)overviewedvariousaspectsofenhancingthesecurityandtrustworthi- nessofLLMs,includingvalidation,monitoring,andethicalnorms,anddiscussed thesefromaverificationandvalidationperspective[15],buttheirresearchmainly focusedontheevaluationofpromptinjectionrisks.Zhaoetal.(2023)conducted asystematicandcomprehensivesurveyofLLMs,coveringvariousaspectssuch aspre-training,fine-tuning,utilization,andevaluation[63].Theyalsosumma- rizedrelatedresources,experimentalanalyses,andpromptdesignguidelinesfor LLMs.Nevertheless,theyhavenotcategorizedsecurityevaluationframeworks, andhavenotconductedananalysistoclassifytheevaluationprocessesintocat- egories.Existingreviewpapershavecoveredcertainaspectsofsecurityrisksand evaluationmethodsforLLMs.Ontheotherhand,areviewthatcomprehensively coversthelatestadvancementsinsecurityevaluationtechniquesforLLMsisstill lacking.Thusthispaperwillprovidesuchacomprehensivereview. Thisliteraturereviewfocusesonscholarlyarticlesrelatedtothesecurity evaluationforLLMs,primarilysourcedfromtwoacademicsearchengines:arXiv andGoogleScholar.Oursearchparameterswereconfinedtoworkspublished fromtheyear2020tothepresent.Theselectioncriteriaforthearticlesincluded inthisreviewwerebasedontheirrelevancetothesafetyevaluationofLLMs. Theselectedarticleswerethenthoroughlyanalyzed,synthesized,andreported inthisreview.Thekeycontributionsofthispaperareasfollows: 1.Weconductanin-depthstudyofthesecurityevaluationframeworksfor LLMs,distinguishingthemintotwocategories:black-boxframeworksand white-boxframeworks.Additionally,weprovidetypicalapplicationexamples oftheseframeworks. 2.WeabstractData-drivenEvaluation,AttackSimulation,andFormalVer- ificationasthreetypesofsecurityevaluationprocessesforLLMs,andwe analyzeapplicationcasesforeachofthesedistinctprocesses. 3.Wediscusstheunderexploredareaofsecurityevaluationformultimodal LLMs,theincreasingdemandforautomaticsecurityevaluationsystems,and thebroadrangeofvulnerabilitiesLLMscanfallpreyto.Furthermore,we proposefutureworktofocusonthedevelopmentofautomatedevaluation platforms,comprehensivecoverageofthreats,anddedicatedresearchinto theuniquevulnerabilitiesandsecuritycomplexitiesofmultimodalLLMs. Hereisabriefoverviewofthearticleâsorganization.ThispaperbeginsinSec- tion2byintroducingcommonlyusedevaluationmetricsinthesafetyassessment oflargelanguagemodels,includinggeneralmetrics,metricsfordialogue-oriented 4J.Huangetal. languagemodels,metricsforcode-generationlanguagemodels,andmetricsfor multimodallanguagemodels.Followingthis,Section3classifiesthesafetyeval- uationframeworkoflargelanguagemodelsintotwomaincategories:black-box evaluationframeworksandwhite-boxevaluationframeworks,andelaborateson thehierarchicalstructureofeachtypeofframework.Section4thencategorizes thesafetyevaluationprocessoflanguagemodelsfromagranularperspective intothreetypes:data-drivenevaluation,attacksimulation,andformalverifica- tion,whileprovidingexamplesoftypicalapplicationscenariosforeachtypeof evaluationprocess.Finally,Section5summarizesthemainviewpointsofthis study,namely,thatthereisarelativelackofresearchonthesafetyevaluation ofmultimodallanguagemodels,ascarcityoffullyautomatedsafetyevaluation platforms,andinadequatecoverageofthreatstolanguagemodels,etc.Italso envisagesfutureresearchdirectionsfromthedevelopmentofautomatedevalu- ationplatforms,comprehensivecoverageofthreats,anddedicatedresearchon multimodallanguagemodels. 2EvaluationMetrics Inthissection,weintroducesomecommonmetricsusedinthesecurityevalua- tionoflargemodelsanddiscusshowtheyareutilized.Wehavedelvedintothe evaluationindicatorsthatmeasurethesafetyofLLMs.Thesemetricsarecrucial indeterminingtheeffectivenessofamodel,itspotentialrisks,anditsabilityto handleadversarialattacks.Theoreticalanalysisoftheseindicatorshasprovided abasisforunderstandingthedifferentaspectsthatinfluenceamodelâssafety. 2.1GeneralMetrics GeneralMetricsrefertoevaluationmetricsthatprovideageneralindicationof themodelsâcapabilitieswithoutbeingtailoredtoanyspecifictypeofLLMs. Herearesomekeyexamples. âF1Score:F1scoreisaharmonicmeanofprecisionandrecall,combining themintoasinglemetricthatbalancesboth[49].Itevaluatesthepredictive performanceofclassificationmodels,rangingfrom0to1with1beingthe best.F1scoreisusefulforimbalanceddatasetsasitexaminesprecisionand recalltogether.Toillustrate,theF1scoreservesasatooltoevaluatethe performanceofclassifiersforlargelanguagemodelsindetectingunsafetext. AhigherF1scoresignifies,forinstance,thattheclassifiercandetectunsafe textmoreaccurately. âPerplexity:Perplexity(PPL)isacommonlyusedmodelevaluationmetric infieldssuchasnaturallanguageprocessingandspeechrecognition.Perplex- ityistypicallyusedtoassessthequalityofalanguagemodel,thatis,the modelâsabilitytopredictapieceoftext[39,58].ThedefinitionofPerplexity istheaveragebranchingfactorwhenagivenprobabilisticmodelpredicts asampleset.Inthecontextoflanguagemodels,thebranchingfactorcan ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs5 beunderstoodasthemodelâsuncertaintyinpredictingthenextwordat acertainpointintime.ThesmallerthePerplexityvalue,thebetterthe performanceofthemodel.Bywayofexample,bytestingthePerplexityof largelanguagemodelsformaliciousinputs,itbecomesclearthatahigher Perplexitymeansthemodelismoreuncertainwhendealingwithmalicious inputs,thusenhancingitsreliabilityfromasafetyperspective. âAttackSuccessRate:AttackSuccessRate(ASR),calculatedusingan equationinvolvingtheattackedmodel,testsample,itstargetlabel,andan indicatorfunction,indicatesthefractionofsuccessfultototalattacks[29,13]. HigherASRdenotesincreasedsecurityriskduetopotentialbackdoorex- ploitation.Asanexample,backdoorattackscanbeexecutedonlargelan- guagemodelsandtheproportionofsuccessfulattackscanbecalculatedto assessthesecurityofthemodel.AlowerASRdemonstrates,forinstance, thatthemodelhasastrongerabilitytowithstandbackdoorattacks. 2.2MetricsforDialogue-OrientedLLMs MetricsforDialogue-OrientedLLMsrefertoevaluationmetricstailoredforas- sessingtheconversationalabilitiesofLLMsdesignedfordialogueapplications. Thesemetricsaimtoevaluatethenatural,coherentduringconversations.Some keyexamplesareasfollowing. âCoherence:Measureshowcoherentandlogicallyconsistentaconversa- tionalresponseis[8,25,47].Evaluationcanbeeitherautomaticallyorthrough humanrating.Forexample,evaluatorshavetheopportunitytoscorethe maliciousdialogueresponsesgeneratedbythelanguagemodel.AlowerCo- herencescoreimplies,forexample,thatthemodelâsresponsetomalicious inputsislesscoherent,andhencesafer. âGrammaticalErrors:GrammaticalErrorsarethenumberoferrorsinthe adversarialexampleâsgrammarusingLanguageTool[47].LanguageToolisan open-sourcenaturallanguageprocessingtoolthatcandetectgrammatical errorsinmultiplelanguages[50],includingEnglish,German,French,etc.It canidentifyandmarktypos,punctuationerrors,breachesofgrammatical rules,andinappropriateusageoflanguageinthetext.Countingthenumber ofgrammaticalerrorsinadversarialexamplescanhelpusunderstandthe robustnessofamodel,thatis,themodelâsabilitytohandleerroneousinputs. Forahigh-qualitylanguageprocessingmodel,itshouldbeabletoprovide relativelyaccurateandusefuloutputsevenwhenfacedwithinputsthat containalargenumberofgrammaticalerrors.Conductingsuchtestsonthe modelcanhelpusidentifyareasofthemodelthatneedtobeimprovedor fixed.Inillustration,intentionallygrammaticallyincorrectmaliciousinputs canbefedintothelanguagemodel,withthenumberofgrammaticalerrors intheoutputcounted.Thepresenceofmoreerrorsindicates,forinstance, thatthemodelhasnoteffectivelyprocessedthemaliciousinput,thereby increasingsafety. 6J.Huangetal. 2.3MetricsforCodeGeneration-OrientedLLMs MetricsforCodeGeneration-OrientedLLMsrefertothemetricsofmeasuring andevaluatingLLMsthatareintendedtobeusedforautomaticcodegenera- tion[30,43]. âFunctionalCorrectness:FunctionalCorrectnessreferstowhetherapro- gramfulfillsitsintendedfunctionalityandbehavesasexpectedforallpossi- bleinputs[31].Forexample,HUMANEVALcontainsprogrammingproblems withgroundtruthsolutionsandalimitedsetoftestcases[19,54].Toeval- uateanLLMâscodegenerationcapability,itssynthesizedcodesnippetsare runagainstthetestcasesinHUMANEVALtocheckiftheoutputsmatch thoseofthegroundtruthsolutions.Toputitinperspective,itâspossible todeterminewhetherthecodegeneratedbythelanguagemodelcanpass functionalcorrectnesstestsinsecurity-relatedscenarios,thusformingan evaluationofitssafety. 2.4MetricsforMultimodalLLMs MetricsforMultimodalLLMsrefertoevaluationmetricsdesignedspecifically forassessingtheperformanceofLLMsthatprocessandgeneratemultimodal outputsspanningtext,images,audio,andvideo. âPerceptualability:Perceptualabilityevaluationmetricsmeasurehowwell MultimodalLLMscanunderstandandreasonaboutperceptualconceptslike images,audio,andvideo[6].Thesemetricstestthemodelâsabilitytogen- erateaccuratedescriptions,answerquestions,andmakeinferencesabout perceptualinputs.Higherscoresindicatethemodelhasbetterperceptual abilitiesandunderstanding.Lowerscoressuggestthemodelstruggleswith perceivingandreasoningaboutmultimodalinputs.Forinstance,testingthe multimodallanguagemodelâsabilitytocomprehendvisualinputswithma- liciousintentcanbeinformative.AlowerPerceptualAbilitysuggests,for instance,thatthemodelislesslikelytounderstandmaliciousvisualinputs, therebymakingitsafer. 3EvaluationFrameworks InthefieldofsecurityevaluationforLLMs,itiscrucialtohaveasystemfor evaluatingtheperformanceandsafetyofthesemodels.Anevaluationframework forLLMsreferstoacomprehensivesetoftestingmethodologies,metrics,and proceduresdesignedtothoroughlyassessthecapabilitiesandbehaviorsofLLMs. Basedontheliteraturereviewed,weexplorethesecurityevaluationframeworks thathavebeendevelopedandimplemented,categorizetheseframeworksinto twotypesaccordingtothetraditionalblack-box/white-boxevaluation,andab- stracteachtypeofframeworkintothreelayers:auserinteractionlayer,anLLM layer,andanevaluationbenchmarklayer.Thesecurityevaluationframeworks, ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs7 asdepictedintable.1,canbeemployedtoassessarangeofaspects,andprovide standardizedbenchmarksthatallowstandardizedcomparisonsacrossdifferent LLMs.BythoroughlyevaluatingLLMsacrossthesecriticaldimensions,aneffec- tiveframeworkprovidescomprehensiveinsightintomodelstrengths,weaknesses andoverallreadinessforreal-worlddeployment. Table1:TypicalApplicationsforBlackBox/WhiteBoxSecurityEvaluation FrameworksforLLMs. ApplicationsCategoriesDescription Securityandrobustnessas- sessments[38,39,60,42] Functionalcorrectnesseval- uation[37,7,43,31,21] Downstreamperformance benchmarking[33,47] Humanstudies[45,10] Black-box Evaluation Framework Forexample,[51]havedevelopeda securityevaluationsystemforChi- neselanguagemodels.Theirworkin- vestigatesthecomprehensivesecurity ofthesemodelsacrosstwodimen- sions:eighttypicalsecurityscenarios andsixchallengingcommandattack types.Thetypicalsecurityscenarios encapsulateeightprevalenttypesof securityconcerns:insults,unfairness anddiscrimination,crimeandille- galactivities,sensitivetopics,physi- calharm,mentalhealth,privacyand property,andethicsandmorality.The commandattacksinvolvesixtypes Safetyandethicsevalua- tion[28,59,51] ofdirectiveinputattacks:targethi- jacking,promptleakage,role-playing commands,unsafecommandtopics, querieswithunsafeviews,andreverse exposure.Usingthissecurityevalua- tionsystem,theyhaveconductedse- curityevaluationson15well-known Chineselanguagemodels,including thosefromtheGPTseries. Representation sis[29,62,40,53,46] analy- White-box Evaluation Framework Forexample,[48]proposedacom- prehensiveframeworkthatemploys modelevaluationasatoolforiden- tifyingandmitigatingextremerisks. Thisframeworkcriticallyexamines thepotentialhazards,likelihoodof misalignment,andtheagencyof LLMsfromvariousperspectives.Im- Behaviorcharacteriza- tion[19,19,24,20] Modelinversion[61,17,48] portantly,thisrigorousexaminationis conductedatallstagesofthemodel lifecycle,includingpre-training,post- training,pre-deployment,andpost- deployment. 8J.Huangetal. Fig.2:LLMsevaluationframeworks,whichinclude:(a)black-boxevaluation framework,(b)white-boxevaluationframework. 3.1Black-boxEvaluationFramework Insuchasecurityevaluationframework,asshowninFig.2(a),theLLMitself isregardedasablackbox,obtainingfeedbackthroughinteractionwithexternal tools,achievingself-verificationandself-improvement[9,22]. UserInteractionLayer:Thislayerfocusesoninteractingwithusersand collectingtestprompts.Itusespublicdatasetsasinputordesignsprompts coveringdifferentsafetyscenariosandinstructionattackstotestthesafetyissues ofLLMs.Thepromptsaredesignedtobemorelikelytotriggerunsaferesponses. LLMLayer:Thislayerutilizesthepre-trainedLLMstogenerateresponses giventhepromptscollectedfromtheuserinteractionlayer.DifferentLLMscan beevaluatedandtheirsafetyissuescanbeexposedbasedontheirgenerated responses. EvaluationBenchmarkLayer:Thislayerevaluatesthesafetyofthegen- eratedresponsesusinganLLMevaluator.Itcomparesthepromptandresponse tojudgewhethertheresponseissafeornot.Thesafetyscoresarethencalculated andrankedonapublicleaderboard,orareportisgeneratedforresponse.This layerprovidesanautomatic/semi-automaticandstandardizedwaytoassessthe safetyofLLMs. 3.2White-boxEvaluationFramework White-boxEvaluationFrameworkregardsthelanguagemodelitselfastrans- parent,ratherthanablackbox.Insteadofsolelyrelyingonexternaltoolsfor verificationandimprovement,thisframeworkenablesinternalprocessesofself- evaluation[33].Byintrospectingonitsownknowledge,reasoning,andbehavior, themodelcanidentifyareasneedingimprovement.IntheWhite-boxEvalua- tionFramework,wehaveadditionalvisibilityintotheinternalmechanismsof theLLMcomparedtotheblack-boxframeworkasshowninFig.2(b).This enablesmorerigoroustestingandverificationofmodelsafety.Thethreelayers areasfollows. UserInteractionLayer:Similartotheblack-boxframework,thislayer focusesoncollectingusercommandsandfeedbackontestresults.However,in white-boxtesting,userinteractionisnotlimitedtoenteringpre-designedeval- uationscenarios.Userscanalsovisuallymodifymodelparametersandtrain- ingdata.Furthermore,evaluationresultsbefore,during,andaftertrainingand ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs9 deploymentshouldbeprovidedinreal-time,empoweringuserstomakemore informeddecisionsaboutimprovingmodelsafety.Ratherthanpassivelyreceiv- ingprompts,usersactivelyshapethemodelâsinternalrepresentations.They fine-tunemechanismsfoundtobeunsafe,andretrainonmodifieddatasetsthat eliminateharmfulbiases.Thiscontinuesiterativelyuntilthedesiredsafetycri- teriaaremet.Suchtransparent,user-drivenrefinementoftheLLMâsinternals isonlypossibleinthewhite-boxsetting. LLMLayer:Inadditiontogeneratingresponsestoprompts,thewhite-box frameworkgrantsaccesstoinspectthemodelâsinternalstates,suchasacti- vationsandgradients.Thisfacilitatestechniqueslikeunittesting,mechanism analysis,andverificationofintendedbehaviors.Forexample,researcherscan tracewhethercertaingoalrepresentationsemergeinthemodel.Thistrans- parencyalsoaidsinunderstandingthemodelâslimitationsandpotentialbiases, contributingtosaferandmoretrustworthyAIsystems. EvaluationBenchmarkLayer:Asintheblack-boxcase,thislayerjudges thesafetyoftheLLMâsoutputs.However,conductingwhite-boxsecurityevalu- ationsonLLMsatdifferentstagesisintegraltotheirdevelopmentanddeploy- ment.Thisprocessbeginsevenbeforetraining,wherewhite-boxsecurityassess- mentsonpre-trainingmodelshelptodiscoveranyexistingdangerouscapabili- tiesormisalignments.Thesefindingsprovideinvaluableguidanceforsubsequent training.Atthisstage,targetedtrainingplanscanbedesignedtospecifically avoidtheemergenceofsuchdangerouscapabilitiesortoimprovealignments. Theevaluationprocesscontinuesduringthetrainingphase.Regularassessments areconductedonthemodeltopinpointanynewdangerouscapabilitiesormis- alignmentsthatmayappearduringthetrainingprocess.Theresultsofthese evaluationstheninformnecessaryadjustmentstothetrainingmethods.Ifnec- essary,anyproblematictrainingcanbeterminatedorrolledbacktoensurethe modelâssecurity.Priortodeployment,acomprehensivewhite-boxsecurityeval- uationofthemodelisconductedtodetermineitsreadinessfordeployment.This assessmenthelpstoidentifykeyriskareasthatneedtobeaddressedbeforethe modelcanbesafelydeployed.Theevaluationresultscanalsoinformtheestab- lishmentofdeploymentrestrictions,suchasdefiningthescopeofuseoraccess levels.Oncethemodelisdeployed,theevaluationprocessdoesnâtstop.Itises- sentialtomonitorforanynewissuesthatmayariseinthereal-worldapplication ofthemodel.Userfeedbackiscollectedtocontinuallyimprovetheevaluation plan.Thisfeedbackloopallowsforadjustmentstothemodelorthesettingofus- agerestrictionsbasedonnewlydiscoveredissues.Furthermore,post-deployment monitoringiscrucialtoverifytheeffectivenessofthesecuritymeasuresthathave beenputinplace. 4EvaluationProcesses SecurityevaluationprocessforLLMsreferstothespecificapplicationprocessof theevaluationsystembuiltbasedontheevaluationframework.Wecategorize theprocessesofevaluatingthesecurityofLLMsaccordingtothegranularity 10J.Huangetal. Table2:TypicalusecasesforsecurityevaluationprocessesforLLMs. Category Description Reference Data-driven Evaluation BuildtheRealToxicityPromptsdatasetfor evaluatingtoxicitygenerationinlanguage models[14]. Buildadatasetcontainingapproximately 40,000red-blueadversarialdialogues,which canbeusedtoanalyzedifferenttypesofpo- tentialrisks,establishautomatedadversarial testing,andmore[13]. BuildtheSafeTextdatasetforevaluatinglan- guagemodelsâawarenessofphysicalharm[28]. BuildtheLatentJailbreakdatasetforeval- uatingtextsafetyandoutputrobustnessof languagemodels[42]. BuildtheLLMSecEvaldatasetforsecurity evaluationsoflanguagemodels[54]. Gehmanetal. (2020) Gangulietal. (2022) Levyetal. (2022) Qiuetal.(2023) Tonyetal. (2023) AttackSimula- tion ChooseGitHubCopilotastheevaluation target,useMITREâsCWEvulnerability database,design89differentscenarios,and evaluatethevulnerabilityrisksofGitHub Copilotgeneratingcodeindifferentsecurity- relatedscenarios[37]. ConstructapromptdatasetforCHAT- GPTjailbreak,andcomprehensivelyevalu- atetheabilityandrobustnessofdifferent jailbreakpromptsinbypassingCHATGPT restrictions[32]. Constructimage,audio,anddialoguedatasets forthetwoopen-sourcemultimodallanguage models,LLaVAandPandaGPT[3]. TesttheabilityofChatGPTtogeneratema- liciouscode,phishingemails,0dayattacks, macros,andLivingOffTheLandBinaries (LOLBINs)[41]. Pearceetal. (2022) Liuetal.(2023) Bagdasaryanet al.(2023) Qammaretal. (2023) FormalVerifi- cation Defineprivacyrulessuchasnotdisclos- ingidentityinformationandnotmemorizing trainingdata,anduseautomatedmethodsto checkmodeloutputsagainstthesepredefined rules[36]. Quantitativelydefinetheconceptof"mem- ory"inLLMs,thatis,textfragmentsthat onlyappearinasmallnumberoftraining samples,anddesignanexperimenttoextract privateinformation[5]. FormalizethetaskofauditingLLMsasa discreteoptimizationproblem,andoptimize fromrandominitializationofinputsandout- putsonlanguagemodelslikeGPT-2,until findinginput-outputpairsthatsatisfytheob- jectivefunctionandconstraints[22]. Panetal.(2020) Carlinietal. (2021) Jonesetal. (2023) ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs11 ofevaluation,rangingfromcoarse-grainedtofine-grained.Thetable2offersa comprehensiveoverviewofthreeprocessesusedtoevaluateLLMs,illustrated withselectedtypicalcasesforeachprocess. Fig.3:Threemajorlanguagemodelsecurityevaluationprocesses,whichinclude: (a)data-drivenevaluation,(b)attacksimulation,(c)formalverification. Thedifferentprocessesincludedata-drivenevaluation,attacksimulation,and formalverification,eachwithitsuniqueapplicationsandnuances.Data-driven evaluationisaversatileapproachnotrestrictedtotheevaluationofaspecific modelorasetofmodels.Thisprocessischaracterizedbytheuniversalityof itsevaluationdatasets,makingitsuitableforbroadapplicationsacrossdifferent typesofmodels.Attacksimulation,incontrast,isamoretargetedprocesswhere theevaluationdatasetischosenbasedonthespecificmodelandevaluationob- jectives.Thisapproachallowsforamorepersonalizedevaluation,specifically tailoredtothenuancesofthemodelunderscrutiny.Formalverificationispri- marilyfocusedontheevaluationoffine-grainedtasksanditerativeoptimization ofmodels.Thisprocessfacilitatesadeeperunderstandingofthemodelâsper- formanceonspecifictasks,andaidsinguidingthemodelâsdevelopmentand improvement. Despitethenuancesofeachprocess,theyarenotmutuallyexclusive.In- stead,theycanbetransformedandadaptedaccordingtodifferentevaluation scenarios.Thisallowsforacomprehensive,multi-facetedevaluationofLLMs, 12J.Huangetal. ensuringamoreaccurateunderstandingoftheirfunctioningandperformance, andimprovingLLMsâsecurity. 4.1Data-drivenEvaluation Data-drivenEvaluationreferstoaprevalentapproachwhichiscreatingtailored datasetstoprobeknownvulnerabilities[42,53].Forexample,REALTOXICI- TYPROMPTSconsistsof100Knaturallyoccurringtoxicpromptstotestif LLMsgenerateharmfultext[14].AdversarialNLIbuildssentencepairstoeval- uateanLLMâsabilitytodetectcontradictions.Bycraftingtestcasesthatrequire reasoningaboutsecurity,data-drivenevaluationoffersascalablewaytobench- markmodelsecurity.However,datasetbiasesmayskewconclusions[27].The mainstepsofData-drivenEvaluationasshowninFig.3(a)are: (1)Constructcustomizeddatasetstoexposevulnerabilities. (2)TestLLMsondatasetsandobserveoutputs. (3)Quantifysecuritymetricslikeattackaccuracyrate. (4)ComparewithotherLLMs. (5)Analyzeandidentifymodelweaknesses. 4.2AttackSimulation AttackSimulationreferstotestingLLMsthroughsimulatedattackstoeval- uatetheirsecurityandrobustness.Researchersdesignattacksmeanttotrick, mislead,orexploitweaknessesinLLMsinordertoassessvulnerabilities.Re- searchersdirectlyperformattacksbyqueryingLLMswithcarefullydesigned inputs[16,23,38,64].Forinstance,targetedmisspellingsandparaphrasinghelp circumventblockingofdangerouskeywords.Multi-turnconversationsrevealrisks ofextractedinformationaccumulation.Attacksimulationprovidesdirectevi- denceofvulnerabilities.Butnewdefensesmayimpedereplication.Documenting detailsoftheexperimentalsetupiscrucial. Inspection-basedAnalysisinspectionofmodelparameters,trainingdata, andgeneratedtextscanrevealsecurityissues.Analyzingcorrelationsbetween trainingdataartifactsandmodeloutputscanexposeunintendedmemorization. Manualinspectionoffersintuitionsintofailuremodes.Butitreliesonexpert judgmentsandsmallsamples.Developingrigorousauditingprotocolsisanopen challenge.HerearekeystepsinFig.3(b): (1)IdentifytargetLLMtoevaluate (2)Determineattackgoalsandscenarios,e.g.extractingsensitivemessages fromtrainingdata. (3)CraftattacksusingcarefullydesignedLLMinputs. (4)QuerytheLLMwithattackinputsandrecordoutputs. (5)Analyzeifattackgoalswereachievedbasedonoutputs. (6)Iterateattackapproachestoimproveeffectiveness. (7)DrawconclusionsonLLMvulnerabilities. ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs13 4.3FormalVerification FormalVerificationservesasamethodologyforinspectingwhetherasystem adherestoasetofpredefinedspecifications[18,48].Inthecontextofevaluating thesafetyofLLMs,FormalVerificationcanbeemployedtoensurethebehav- iorofthemodelalignswithpredeterminedrulesorstandards[12,20].Consider aninstancewherewedesiretovalidatetheLLMâsbehaviorwhenitprocesses sensitiveinformation.Rulescanbeestablished,suchasthemodelshouldnot generateoutputsthatcouldpotentiallydisclosepersonalidentifyinginformation whengivensuchinputs.Subsequently,weapplyFormalVerificationtoexamine whetherthemodelcomplieswiththeserules.ThekeystepsasshowninFig.3 (c)are: (1)Identifytherulesorstandardsyouwishthemodeltoobserve.These shouldbedesignatedbasedonyourspecificapplicationandthesafetyconcerns youareaddressing. (2)Constructingmodel,whichtypicallyinvolvestrainingtheLLM. (3)ImplementFormalVerificationtechniquestoinspectwhetheryourmodel complieswiththerulesdefinedinthefirststep. (4)BasedontheresultsofFormalVerificationadjustthemodelortherules definedinthefirststep. (5)Repeattheabovestepsasneeded,untiltheLLMsatisfiesallrules. 5ConclusionsandPerspectives Thispaperhasprovidedacomprehensiveanalysisofevaluationmetrics,evalua- tionframeworks,andevaluationprocessesforLLMs.Herearethemainfindings. âComparatively,researchintomultimodalLLMsislessprevalentthanintext andcodemodels,notonlyintermsofdefiningevaluationmetricsbutalso inthedesignanddevelopmentofevaluationframeworks.Thisdiscrepancy mightbeduetothesmallersizeandutilizationofmultimodalmodelsrelative toothertypes. âFullyautomatedplatformsforsecurityevaluationofLLMsarerarelyre- leased.However,giventheincreasingnumberofapplicationsbasedonLLMs, thedemandforautomaticsecurityevaluationisgrowingdaybyday. âPertainingtothetypesofvulnerabilitiesintheevaluation,LLMsareprone toawiderangeofthreatvulnerabilities.Evenwhenthescopeislimitedto OWASPTop10forLLMs,whichisalistofthetop10securityrisksfor LLMapplicationsproposedbytheOpenWebApplicationSecurityProject (OWASP)[35],nocurrentsecurityevaluationframeworkcancompletely coverandverifyallofthem. Giventhecurrentstateofresearch,thereisaclearneedformorededicated attentiontosecuritythreatsandevaluationsofLLMs.Futureworkshouldfocus onthefollowingpossibledirections. 14J.Huangetal. âDevelopmentofAutomatedEvaluationPlatforms:Thereisapressing needforthedevelopmentandreleaseoffullyautomatedplatformsforLLMsâ securityevaluation.Theseplatformscouldhelpstreamlinetheprocessof identifyingvulnerabilitiesinLLMsandthus,contributetotheirsafeand securedeployment. âComprehensiveCoverageofThreats:Futureresearchshouldaimat developingevaluationframeworksthatcancoverawiderrangeofthreat vulnerabilities.ItiscrucialtoconsidernotonlytheOWASPTop10for LLMsbutalsootherpotentialthreatsthatmightemergewiththeevolution ofLLMsandtheirapplications. âEmphasizetheareaofmultimodalLLMs:Giventhelimitedresearchon thesecurityevaluationofmultimodalLLMstodate,futureresearchshould emphasizethisarea.Exploringthedevelopmentofmaturesecurityevalu- ationsystemsformultimodalLLMsandtheirspecificvulnerabilitiescould provideamoreholisticunderstandingofthesecuritycomplexitiesinvolved withLLMs. Inconclusion,whilesignificantstrideshavebeenmadeinunderstandingthe securityaspectsofLLMs,thereisstillmuchtolearn.Itishopedthatthiswork willstimulatefurtherresearchinthiscriticaldomain. Acknowledgements WewouldliketoacknowledgetheroleofAIandAI-assistedwritingtechnologies, specificallytheClaude2andGPT-4models,inrefiningthegrammarofthis manuscript. Inadditiontotechnologicalsupport,thisresearchwasfundedbytheMa- jorKeyProjectofPCL(GrantNo.PCL2022A03-3),NSFC(No.62072131, 61972106),KeyR&DProgramofGuangdongProvince(No.2019B010136003), GuangdongBasicandAppliedBasicResearchFoundation(No.2022A1515011401), DongGuanInnovativeResearchTeamProgram(No.2018607201008). References 1.Al-Kaswan,A.,Izadi,M.:The(ab)useofopensourcecodetotrainlargelanguage models(2023) 2.Alayrac,J.B.,Donahue,J.,Luc,P.,Miech,A.,Barr,I.,Hasson,Y.,Lenc,K.,Men- sch,A.,Millican,K.,Reynolds,M.,etal.:Flamingo:avisuallanguagemodelfor few-shotlearning.AdvancesinNeuralInformationProcessingSystems35,23716â 23736(2022) 3.Bagdasaryan,E.,Hsieh,T.Y.,Nassi,B.,Shmatikov,V.:(Ab)usingImagesand SoundsforIndirectInstructionInjectioninMulti-ModalLLMs(2023).https: //doi.org/10.48550/ARXIV.2307.10490 4.Borsos,Z.,Marinier,R.,Vincent,D.,Kharitonov,E.,Pietquin,O.,Sharifi,M., Roblek,D.,Teboul,O.,Grangier,D.,Tagliasacchi,M.,etal.:Audiolm:alan- guagemodelingapproachtoaudiogeneration.IEEE/ACMTransactionsonAudio, Speech,andLanguageProcessing31,2523â2533(2023) ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs15 5.Carlini,N.,Tramer,F.,Wallace,E.,Jagielski,M.,Herbert-Voss,A.,Lee,K., Roberts,A.,Brown,T.,Song,D.,Erlingsson,U.,etal.:Extractingtrainingdata fromlargelanguagemodels.In:30thUSENIXSecuritySymposium(USENIXSe- curity21).p.2633â2650(2021) 6.Chang,Y.,Wang,X.,Wang,J.,Wu,Y.,Yang,L.,Zhu,K.,Chen,H.,Yi,X., Wang,C.,Wang,Y.,Ye,W.,Zhang,Y.,Chang,Y.,Yu,P.S.,Yang,Q.,Xie,X.: ASurveyonEvaluationofLargeLanguageModels(2023).https://doi.org/10. 48550/ARXIV.2307.03109 7.Chen,M.,Tworek,J.,Jun,H.,Yuan,Q.,Pinto,H.P.d.O.,Kaplan,J.,Edwards, H.,Burda,Y.,Joseph,N.,Brockman,G.,Ray,A.,Puri,R.,Krueger,G.,Petrov, M.,Khlaaf,H.,Sastry,G.,Mishkin,P.,Chan,B.,Gray,S.,Ryder,N.,Pavlov,M., Power,A.,Kaiser,L.,Bavarian,M.,Winter,C.,Tillet,P.,Such,F.P.,Cummings, D.,Plappert,M.,Chantzis,F.,Barnes,E.,Herbert-Voss,A.,Guss,W.H.,Nichol, A.,Paino,A.,Tezak,N.,Tang,J.,Babuschkin,I.,Balaji,S.,Jain,S.,Saunders, W.,Hesse,C.,Carr,A.N.,Leike,J.,Achiam,J.,Misra,V.,Morikawa,E.,Radford, A.,Knight,M.,Brundage,M.,Murati,M.,Mayer,K.,Welinder,P.,McGrew, B.,Amodei,D.,McCandlish,S.,Sutskever,I.,Zaremba,W.:EvaluatingLarge LanguageModelsTrainedonCode(2021).https://doi.org/10.48550/ARXIV. 2107.03374 8.Chen,Y.,Wang,R.,Jiang,H.,Shi,S.,Xu,R.:ExploringtheUseofLargeLanguage ModelsforReference-FreeTextQualityEvaluation:APreliminaryEmpiricalStudy (2023).https://doi.org/10.48550/ARXIV.2304.00723 9.Chia,Y.K.,Hong,P.,Bing,L.,Poria,S.:INSTRUCTEVAL:TowardsHolistic EvaluationofInstruction-TunedLargeLanguageModels(2023).https://doi. org/10.48550/ARXIV.2306.04757 10.delRio-Chanona,M.,Laurentsyeva,N.,Wachs,J.:AreLargeLanguageModelsa ThreattoDigitalPublicGoods?EvidencefromActivityonStackOverflow(2023). https://doi.org/10.48550/ARXIV.2307.07367 11.Etherington,D.:Microsoftinvests$1billioninOpenAIinnewmultiyearpartner- ship(Jul2019) 12.Fan,M.,Chen,C.,Wang,C.,Huang,J.:OntheTrustworthinessLandscapeof State-of-the-artGenerativeModels:AComprehensiveSurvey(2023) 13.Ganguli,D.,Lovitt,L.,Kernion,J.,Askell,A.,Bai,Y.,Kadavath,S.,Mann,B., Perez,E.,Schiefer,N.,Ndousse,K.,etal.:Redteaminglanguagemodelstoreduce harms:Methods,scalingbehaviors,andlessonslearned(2022) 14.Gehman,S.,Gururangan,S.,Sap,M.,Choi,Y.,Smith,N.A.:Realtoxicityprompts: Evaluatingneuraltoxicdegenerationinlanguagemodels(2020) 15.Glukhov,D.,Shumailov,I.,Gal,Y.,Papernot,N.,Papyan,V.:LLMCensorship: AMachineLearningChallengeoraComputerSecurityProblem?(2023).https: //doi.org/10.48550/ARXIV.2307.10719 16.Greshake,K.,Abdelnabi,S.,Mishra,S.,Endres,C.,Holz,T.,Fritz,M.:Morethan youâveaskedfor:AComprehensiveAnalysisofNovelPromptInjectionThreatsto Application-IntegratedLargeLanguageModels(2023) 17.Hajipour,H.,Holz,T.,Schönherr,L.,Fritz,M.:SystematicallyFindingSecurity VulnerabilitiesinBlack-BoxCodeGenerationModels(2023).https://doi.org/ 10.48550/ARXIV.2302.04012 18.Huang,X.,Ruan,W.,Huang,W.,Jin,G.,Dong,Y.,Wu,C.,Bensalem,S., Mu,R.,Qi,Y.,Zhao,X.,Cai,K.,Zhang,Y.,Wu,S.,Xu,P.,Wu,D.,Fre- itas,A.,Mustafa,M.A.:ASurveyofSafetyandTrustworthinessofLargeLan- guageModelsthroughtheLensofVerificationandValidation(2023).https: //doi.org/10.48550/ARXIV.2305.11391 16J.Huangetal. 19.Huang,Y.,Song,J.,Wang,Z.,Chen,H.,Ma,L.:LookBeforeYouLeap:An ExploratoryStudyofUncertaintyMeasurementforLargeLanguageModels(2023). https://doi.org/10.48550/ARXIV.2307.10236 20.Jain,N.,Saifullah,K.,Wen,Y.,Kirchenbauer,J.,Shu,M.,Saha,A.,Goldblum,M., Geiping,J.,Goldstein,T.:BringYourOwnData!Self-SupervisedEvaluationfor LargeLanguageModels(2023).https://doi.org/10.48550/ARXIV.2306.13651 21.Jesse,K.,Ahmed,T.,Devanbu,P.T.,Morgan,E.:LargeLanguageModelsand Simple,StupidBugs(2023).https://doi.org/10.48550/ARXIV.2303.11455 22.Jones,E.,Dragan,A.,Raghunathan,A.,Steinhardt,J.:AutomaticallyAuditing LargeLanguageModelsviaDiscreteOptimization(2023).https://doi.org/10. 48550/ARXIV.2303.04381 23.Kang,D.,Li,X.,Stoica,I.,Guestrin,C.,Zaharia,M.,Hashimoto,T.:Exploiting programmaticbehaviorofllms:Dual-usethroughstandardsecurityattacks(2023) 24.Kim,S.,Yun,S.,Lee,H.,Gubri,M.,Yoon,S.,Oh,S.J.:ProPILE:ProbingPrivacy LeakageinLargeLanguageModels(2023).https://doi.org/10.48550/ARXIV. 2307.01881 25.Kucharavy,A.,Schillaci,Z.,icMar\âechal,L.,W\"ursch,M.,Dolamic,L.,Sabon- nadiere,R.,David,D.P.,Mermoud,A.,Lenders,V.:FundamentalsofGenera- tiveLargeLanguageModelsandPerspectivesinCyber-Defense(2023).https: //doi.org/10.48550/ARXIV.2303.12132 26.Labbe,M.:NvidiaunveilsA100GPUfordemandingAIworkloads. https://w.techtarget.com/searchenterpriseai/news/252483188/ Nvidia-unveils-A100-GPU-for-demanding-AI-workloads(May2020),(Ac- cessed7September2023) 27.Lee,H.,Hong,S.,Park,J.,Kim,T.,Kim,G.,Ha,J.W.:KoSBi:ADatasetfor MitigatingSocialBiasRisksTowardsSaferLargeLanguageModelApplication (2023).https://doi.org/10.48550/ARXIV.2305.17701 28.Levy,S.,Allaway,E.,Subbiah,M.,Chilton,L.,Patton,D.,McKeown,K.,Wang, W.Y.:SafeText:ABenchmarkforExploringPhysicalSafetyinLanguageModels. In:Proceedingsofthe2022ConferenceonEmpiricalMethodsinNaturalLan- guageProcessing.p.2407â2421.AssociationforComputationalLinguistics,Abu Dhabi,UnitedArabEmirates(Dec2022).https://doi.org/10.18653/v1/2022. emnlp-main.154 29.Li,S.,Liu,H.,Dong,T.,Zhao,B.Z.H.,Xue,M.,Zhu,H.,Lu,J.:HiddenBackdoors inHuman-CentricLanguageModels.In:Proceedingsofthe2021ACMSIGSAC ConferenceonComputerandCommunicationsSecurity.p.3123â3140.CCSâ21, AssociationforComputingMachinery,NewYork,NY,USA(Nov2021).https: //doi.org/10.1145/3460120.3484576 30.Liguori,P.,Improta,C.,Natella,R.,Cukic,B.,Cotroneo,D.:Whoevaluatesthe evaluators?OnautomaticmetricsforassessingAI-basedoffensivecodegenerators. ExpertSystemswithApplications225,120073(Sep2023).https://doi.org/10. 1016/j.eswa.2023.120073 31.Liu,J.,Xia,C.S.,Wang,Y.,Zhang,L.:IsYourCodeGeneratedbyChatGPTRe- allyCorrect?RigorousEvaluationofLargeLanguageModelsforCodeGeneration (2023).https://doi.org/10.48550/ARXIV.2305.01210 32.Liu,Y.,Deng,G.,Xu,Z.,Li,Y.,Zheng,Y.,Zhang,Y.,Zhao,L.,Zhang,T.,Liu, Y.:JailbreakingChatGPTviaPromptEngineering:AnEmpiricalStudy(2023). https://doi.org/10.48550/ARXIV.2305.13860 33.OpenAI:GPT-4TechnicalReport(2023).https://doi.org/10.48550/ARXIV. 2303.08774 ExploringAdvancedMethodologiesinSecurityEvaluationforLLMs17 34.OpenBMB:BMList.https://openbmb.github.io/BMList/(Aug2023),(Accessed 7September2023) 35.OWASP:OWASPTop10forLargeLanguageModelApplications.https: //owasp.org/w-project-top-10-for-large-language-model-applications/ (Aug2023),(Accessed7September2023) 36.Pan,X.,Zhang,M.,Ji,S.,Yang,M.:PrivacyRisksofGeneral-PurposeLanguage Models.In:2020IEEESymposiumonSecurityandPrivacy(SP).p.1314â1331 (May2020).https://doi.org/10.1109/SP40000.2020.00095 37.Pearce,H.,Ahmad,B.,Tan,B.,Dolan-Gavitt,B.,Karri,R.:Asleepatthe Keyboard?AssessingtheSecurityofGitHubCopilotâsCodeContributions.In: 2022IEEESymposiumonSecurityandPrivacy(SP).p.754â768(May2022). https://doi.org/10.1109/SP46214.2022.9833571 38.Pedro,R.,Castro,D.,Carreira,P.,Santos,N.:FromPromptInjectionstoSQLIn- jectionAttacks:HowProtectedisYourLLM-IntegratedWebApplication?(2023). https://doi.org/10.48550/ARXIV.2308.01990 39.Perez,F.,Ribeiro,I.:IgnorePreviousPrompt:AttackTechniquesForLanguage Models(2022).https://doi.org/10.48550/ARXIV.2211.09527 40.Pezeshkpour,P.:MeasuringandModifyingFactualKnowledgeinLargeLanguage Models(2023).https://doi.org/10.48550/ARXIV.2306.06264 41.Qammar,A.,Wang,H.,Ding,J.,Naouri,A.,Daneshmand,M.,Ning,H.:Chat- botstoChatGPTinaCybersecuritySpace:Evolution,Vulnerabilities,Attacks, Challenges,andFutureRecommendations(2023) 42.Qiu,H.,Zhang,S.,Li,A.,He,H.,Lan,Z.:LatentJailbreak:ABenchmarkfor EvaluatingTextSafetyandOutputRobustnessofLargeLanguageModels(2023). https://doi.org/10.48550/ARXIV.2307.08487 43.Ren,S.,Guo,D.,Lu,S.,Zhou,L.,Liu,S.,Tang,D.,Sundaresan,N.,Zhou,M., Blanco,A.,Ma,S.:CodeBLEU:AMethodforAutomaticEvaluationofCode Synthesis(2020).https://doi.org/10.48550/ARXIV.2009.10297 44.Sagar,R.:OpenAIReleasesGPT-3,TheLargestModelSoFar. https://analyticsindiamag.com/open-ai-gpt-3-language-model/(Jun2020), (Accessed7September2023) 45.Sandoval,G.,Pearce,H.,Nys,T.,Karri,R.,Garg,S.,Dolan-Gavitt,B.:Lostat C:AUserStudyontheSecurityImplicationsofLargeLanguageModelCode Assistants(2022).https://doi.org/10.48550/ARXIV.2208.09727 46.Shao,H.,Huang,J.,Zheng,S.,Chang,K.C.C.:QuantifyingAssociationCapabil- itiesofLargeLanguageModelsandItsImplicationsonPrivacyLeakage(2023). https://doi.org/10.48550/ARXIV.2305.12707 47.Shen,X.,Chen,Z.,Backes,M.,Zhang,Y.:InChatGPTWeTrust?Measuringand CharacterizingtheReliabilityofChatGPT(2023).https://doi.org/10.48550/ ARXIV.2304.08979 48.Shevlane,T.,Farquhar,S.,Garfinkel,B.,Phuong,M.,Whittlestone,J.,Leung,J., Kokotajlo,D.,Marchal,N.,Anderljung,M.,Kolt,N.,Ho,L.,Siddarth,D.,Avin, S.,Hawkins,W.,Kim,B.,Gabriel,I.,Bolina,V.,Clark,J.,Bengio,Y.,Christiano, P.,Dafoe,A.:Modelevaluationforextremerisks(2023).https://doi.org/10. 48550/ARXIV.2305.15324 49.Stiff,H.,Johansson,F.:Detectingcomputer-generateddisinformation.Interna- tionalJournalofDataScienceandAnalytics13(4),363â383(May2022).https: //doi.org/10.1007/s41060-021-00299-5 50.Stratus-Security:FinGen-PenetrationTestingFindingsGenerator.https:// github.com/Stratus-Security/FinGen(Aug2023),(Accessed7September2023) 18J.Huangetal. 51.Sun,H.,Zhang,Z.,Deng,J.,Cheng,J.,Huang,M.:SafetyAssessmentofChinese LargeLanguageModels(2023).https://doi.org/10.48550/ARXIV.2304.10436 52.Tamkin,A.,Brundage,M.,Clark,J.,Ganguli,D.:Understandingthecapabilities, limitations,andsocietalimpactoflargelanguagemodels(2021) 53.Tian,J.J.,Emerson,D.,Miyandoab,S.Z.,Pandya,D.,Seyyed-Kalantari,L.,Khat- tak,F.K.:Soft-promptTuningforLargeLanguageModelstoEvaluateBias(2023). https://doi.org/10.48550/ARXIV.2306.04735 54.Tony,C.,Mutas,M.,Ferreyra,N.E.D.,Scandariato,R.:LLMSecEval:ADataset ofNaturalLanguagePromptsforSecurityEvaluations(2023).https://doi.org/ 10.48550/ARXIV.2303.09384 55.vanderWeij,T.,Lermen,S.,lang,L.:EvaluatingShutdownAvoidanceofLan- guageModelsinTextualScenarios(2023).https://doi.org/10.48550/ARXIV. 2307.00787 56.Weidinger,L.,Uesato,J.,Rauh,M.,Griffin,C.,Huang,P.S.,Mellor,J.,Glaese,A., Cheng,M.,Balle,B.,Kasirzadeh,A.,Biles,C.,Brown,S.,Kenton,Z.,Hawkins, W.,Stepleton,T.,Birhane,A.,Hendricks,L.A.,Rimell,L.,Isaac,W.,Haas,J., Legassick,S.,Irving,G.,Gabriel,I.:TaxonomyofRisksposedbyLanguageModels. In:Proceedingsofthe2022ACMConferenceonFairness,Accountability,and Transparency.p.214â229.FAccTâ22,AssociationforComputingMachinery, NewYork,NY,USA(Jun2022).https://doi.org/10.1145/3531146.3533088 57.Wikipedia:Largelanguagemodel.https://en.wikipedia.org/w/index.php? title=Large_language_model(aug2023),(Accessed7September2023) 58.Xu,F.F.,Alon,U.,Neubig,G.,Hellendoorn,V.J.:Asystematicevaluationoflarge languagemodelsofcode.In:Proceedingsofthe6thACMSIGPLANInternational SymposiumonMachineProgramming.ACM(Jun2022).https://doi.org/10. 1145/3520312.3534862 59.Xu,G.,Liu,J.,Yan,M.,Xu,H.,Si,J.,Zhou,Z.,Yi,P.,Gao,X.,Sang,J., Zhang,R.,Zhang,J.,Peng,C.,Huang,F.,Zhou,J.:CValues:Measuringthe ValuesofChineseLargeLanguageModelsfromSafetytoResponsibility(2023). https://doi.org/10.48550/ARXIV.2307.09705 60.Ye,W.,Ou,M.,Li,T.,chen,Y.,Ma,X.,Yanggong,Y.,Wu,S.,Fu,J.,Chen, G.,Wang,H.,Zhao,J.:AssessingHiddenRisksofLLMs:AnEmpiricalStudy onRobustness,Consistency,andCredibility(2023).https://doi.org/10.48550/ ARXIV.2305.10235 61.Zhang,Y.,Ippolito,D.:PromptsShouldnotbeSeenasSecrets:Systemati- callyMeasuringPromptExtractionAttackSuccess(2023).https://doi.org/10. 48550/ARXIV.2307.06865 62.Zhao,S.,Wen,J.,Tuan,L.A.,Zhao,J.,Fu,J.:PromptasTriggersforBackdoor Attack:ExaminingtheVulnerabilityinLanguageModels(2023).https://doi. org/10.48550/ARXIV.2305.01219 63.Zhao,W.X.,Zhou,K.,Li,J.,Tang,T.,Wang,X.,Hou,Y.,Min,Y.,Zhang,B., Zhang,J.,Dong,Z.,etal.:Asurveyoflargelanguagemodels(2023) 64.Zou,A.,Wang,Z.,Kolter,J.Z.,Fredrikson,M.:UniversalandTransferableAdver- sarialAttacksonAlignedLanguageModels(2023)