½ñÈÕ¹Ù·½Åû¶ÐÐÒµÑо¿Ð§¹ûÃÀý³ÆÃÀÒÁ̸ÅÐÖØ»ØÕý¹ì ºÜÐË·ÜΪÄú½â´ðÕâ¸öÎÊÌ⣬£¬£¬£¬ÈÃÎÒÀ´°ïÄúÏêϸ˵Ã÷һϡ£¡£Æ·ÅÆÊÚȨ±¨Ð޵绰£¬£¬£¬£¬¿ìËÙÉÏÃÅ·þÎñ

¹ðÁÖÊÐÐã·åÇø¸£½¨Ê¡ÆÎÌïÊгÇÏáÇø²ýƽÇøСÌÀɽÕòÁÉÄþÊ¡ÌúÁëÊпªÔ­Êзą́ÇøºÍÒå½ÖµÀ±±³½Çø¹ãÔ´½ÖµÀ¾®ÚêÏزâÓãÕò¹ã¶«Ê¡ÉÇÍ·ÊÐÁúºþÇøÁÉÄþÊ¡ÉòÑôÊн­Î÷Ê¡¸ÓÖÝÊÐʯ³ÇÏØÔÞ»ÊÏØÎ÷ÁúÃÅÏ縣½¨Ê¡ÆÎÌïÊÐÏÉÓÎÏØËÄ´¨Ê¡³É¶¼»áж¼Çø¸ÊËàÊ¡¸ÊÄϲØ×å×ÔÖÎÖݹ㶫ʡ¹ãÖÝÊÐÔ½ÐãÇøºÚÁú½­Ê¡¼¦Î÷Êм¦¹ÚÇø¸£½¨Ê¡¸£ÖÝÊÐÓÀÌ©Ïؽ­Î÷Ê¡¸ÓÖÝÊÐʯ³ÇÏؼªÁÖÊ¡³¤´ºÊÐË«ÑôÇø¹óÖÝÊ¡Í­ÈÊÊÐÓñÆÁ¶±×å×ÔÖÎÏغþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݽ¨Ê¼ÏØÐÂÀÖÊÐЭÉñÏçºÓ¶«Çø´óÖ±¹Á½ÖµÀÏòÑôÇø¹ÜׯµØÇøÃÜÔÆÇøºÓÄÏÕ¯Õò¹ðÁÖÊÐÐã·åÇøÎߺþÊÐÄÏÁêÏظ£½¨Ê¡¸£ÖÝÊÐÂÞÔ´ÏØÄÏÑôÊÐÄÚÏçÏØÏòÑôÇø¶¹¸÷ׯµØÇø·ą́Çø³¤ÐÁµêÕòÃÅÍ·¹µÇø´óÓø½ÖµÀÏòÑôÇøË«¾®½ÖµÀÎÚ³ľÆëÊÐɳÒÀ°Í¿ËÇø¸£½¨Ê¡ÏÃÃÅÊк£²×Çø¾²º£ÇøÎ÷µÔׯÕò¸£½¨Ê¡ÄÏƽÊн¨ê±ÊÐÂåÑôÊÐÈêÑôÏظ£½¨Ê¡ÕÄÖÝÊÐÁúº£Êй㶫ʡÉÇÍ·ÊÐÁúºþÇøËÄ´¨Ê¡ÀÖɽÊиÊËàÊ¡¶¨Î÷ÊÐμԴÏØÔªÊÏÏØËÕ´åÏçÉÂÎ÷Ê¡Î÷°²ÊÐδÑëÇøËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖݸ£½¨Ê¡ÏÃÃÅÊк£²×ÇøÂí°°É½Êв©ÍûÇø½­Î÷Ê¡¾Å½­ÊÐÎäÄþÏØÏòÑôÇøË«¾®½ÖµÀËÄ´¨Ê¡³É¶¼»á½ðÅ£ÇøÄÏ¿ªÇø³¤ºç½ÖµÀ¹ã¶«Ê¡»ÝÖÝÊÐÁúÃÅÏØÁÉÄþÊ¡ÁÉÑôÊа×ËþÇøºÓÎ÷ÇøÌÒÔ°½ÖµÀÇàÐãÇø½­ËÕÊ¡ÄÏͨÊÐÆô¶«Êп¦Ê²µØÇøÒ¶³ÇÏؽ­ËÕÊ¡Á¬ÔƸÛÊиÓÓÜÇø²ýƽÇø»ØÁú¹Û½ÖµÀ·ą́ÇøÍðƽ³ÇµØÇø½­Î÷Ê¡¼ª°²Êмª°²ÏعóÖÝÊ¡Ç­Äϲ¼ÒÀ×åÃç×å×ÔÖÎÖÝ»ÝË®ÏØÖØÇìÊÐÊÐϽÇø±±íÕÇøɽ¶«Ê¡Íþº£ÊйðÁÖÊÐÐã·åÇøºþ±±Ê¡Ò˲ýÊÐïö¹éÏؽ­Î÷Ê¡¼ª°²ÊÐÓÀÐÂÏغÚÁú½­Ê¡¹þ¶û±õÊаÍÑåÏظÊËàʡ¤ÄÏÊÐÎ䶼Çø·ą́Çø³¤ÐÁµêÕò¾®ÚêÏزâÓãÕòɽ¶«Ê¡Ì©°²Êж«Æ½ÏØ´óÐËÇøÖØÇìÊÐÊÐϽÇø±±íÕÇø¸£½¨Ê¡ÄÏƽÊÐɽ¶«Ê¡Íþº£Êпª·âÊÐͨÐíÏؼ»ÖÝÇø¹ÙׯÕòɽ¶«Ê¡ÇൺÊгÇÑôÇø³ç×óÊÐÄþÃ÷Ïع㶫ʡÔƸ¡ÊÐÐÂÐËÏØËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖݵ¤°ÍÏØÐí²ýÊн¨°²ÇøÔÆÄÏÊ¡Å­½­ÀüËÛ×å×ÔÖÎÖݸ£¹±Ïع㶫ʡÉîÛÚÊÐÑÓÇìÇø¿µ×¯Õòƽ¶¥É½ÊÐÕ¿ºÓÇø»³ÈáÇøÁúɽ½ÖµÀÎ÷ÏçÌÁÇøÇÕÖÝÊÐÏòÑôÇø¾ÆÏÉÇŽֵÀ¸£½¨Ê¡¸£ÖÝÊÐÂÞÔ´Ïؽ­ËÕÊ¡ÐìÖÝÊÐÐÂÒÊÊб¦ÛæÇøÅ£¼ÒÅÆÕòÉÂÎ÷Ê¡Î÷°²ÊÐδÑëÇøÉÂÎ÷Ê¡Î÷°²ÊÐδÑëÇøÇຣʡº£Î÷ÃɹÅ×å²Ø×å×ÔÖÎÖݵÂÁî¹þÊиÊËàÊ¡°×ÒøÊо°Ì©Ïغþ±±Ê¡Ò˲ýÊÐïö¹éÏØÎ÷ÏçÌÁÇø

±¾Ô¹ٷ½Åû¶ÐÐÒµÑо¿Ï£ÍûMBench: Ç廪xÌÚѶÁªºÏ½ç˵ÊÓƵÌìÏÂÄ£×ӵĺã¾ÃÓ°ÏóÄÜÁ¦£¬£¬£¬£¬ºÜÐË·ÜΪÄú½â´ðÕâ¸öÎÊÌ⣬£¬£¬£¬ÈÃÎÒÀ´°ïÄúÏêϸ˵Ã÷Ò»ÏÂ:¹Ù·½·þÎñרÏߣ¬£¬£¬£¬Ö§³Ö¶àÆ·ÅƱ¨ÐÞ

ÉϺ£ÊÐÊÐϽÇø¼Î¶¨Çø½­Î÷ʡӥ̶ÊÐÓ཭ÇøÁÉÄþÊ¡ÌúÁëÊпªÔ­ÊÐÏòÑôÇø¾ÆÏÉÇŽֵÀÁÉÄþÊ¡ÁÉÑôÊа×ËþÇøºÓ¶«Çø¶«Ð½ֵÀÁÉÄþÊ¡ÏòÑôÊб±Æ±Êн¹×÷ÊÐÂåÑôÊÐÈêÑôÏØÔÞ»ÊÏØԺͷÕò·À³Ç¸ÛÊж«ÐËÊÐËÄ´¨Ê¡ÀÖɽÊÐɽÎ÷Ê¡ÐÃÖÝÊж¨ÏåÏØËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖÝÑÓÇìÇø¿µ×¯ÕòɽÎ÷Ê¡ÐÃÖÝÊÐÄþÎäÏØÐÂÀÖÊÐЭÉñÏç¹óÖÝÊ¡Ç­¶«ÄÏÃç×嶱×å×ÔÖÎÖÝÌìÖùÏØ¿ª·âÊÐͨÐíÏر±º£Êк£³ÇÇøº£ÄÏÊ¡ÙÙÖÝÊÐɽÎ÷Ê¡Ë·ÖÝÊÐɽÒõÏؼªÁÖÊ¡ËÄƽÊÐÌúÎ÷ÇøÕÔÏØɳºÓµêÕòÖØÇìÊÐÏØÎ×ɽÏØƽɽÏØÕ¬±±Ïçºþ±±Ê¡ÏåÑôÊÐÏåÖÝÇøÎ÷ÇàÇø¾«ÎäÕòÇຣʡÓñÊ÷²Ø×å×ÔÖÎÖÝÖζàÏØÖØÇìÊÐÊÐϽÇø±±íÕÇø½­Î÷Ê¡ÒË´ºÊÐÒË·áÏØɽÎ÷Ê¡½ú¶¼»áÔóÖÝÏØÎäÇåÇøãâ¹Á¸ÛÕòɽÎ÷Ê¡½ú¶¼»á¸ßƽÊÐƽ¶¥É½ÊÐÕ¿ºÓÇøÉÂÎ÷Ê¡ººÖÐÊÐÎ÷ÏçÏؽ­Î÷Ê¡¸ÓÖÝÊÐʯ³ÇÏØå§ÑôÊÐÄÏÀÖÏØÎ÷²Ø°¢ÀïµØÇøÈÕÍÁÏعÌÔ­ÊÐÎ÷¼ªÏØÔÆÄÏÊ¡ºìºÓ¹þÄá×åÒÍ×å×ÔÖÎÖÝãòÎ÷ÏØɽ¶«Ê¡Ì©°²ÊÐ̩ɽÇøÔÆÄÏÊ¡ÓñϪÊÐÐÂƽÒÍ×å´ö×å×ÔÖÎÏØÎ÷³ÇÇøÌìÇŽֵÀ½­Î÷Ê¡¼ª°²ÊÐÓÀÐÂÏØÁÉÄþÊ¡´óÁ¬ÊÐÂÃ˳¿ÚÇø½¹×÷ÊÐÖÐÕ¾ÇøÔªÊÏÏØÒó´åÕòÔÞ»ÊÏØÎ÷ÁúÃÅÏçËÄ´¨Ê¡Á¹É½ÒÍ×å×ÔÖÎÖÝÕѾõÏع㶫ʡ½­ÃÅÊÐÄÚÃɹÅÐË°²ÃËÎÚÀ¼ºÆÌØÊÐÏòÑôÇøСºìÃŵØÇø·ą́ÇøÍðƽ³ÇµØÇøÇå¾²ÇøÄÏÊнֵÀËÄ´¨Ê¡Ò˱öÊн­°²ÏØɽ¶«Ê¡¶«ÓªÊпÑÀûÇø²ýƽÇø»ØÁú¹Û½ÖµÀÉÂÎ÷Ê¡ÏÌÑôÊÐÐËƽÊÐɽ¶«Ê¡×Ͳ©ÊÐÁÙ×ÍÇøɽÎ÷Ê¡³¤ÖÎÊÐÏåÔ«ÏØɽÎ÷Ê¡³¤ÖÎÊÐÏåÔ«ÏØÄÚÃɹÅÎýÁÖ¹ùÀÕÃËÏâ»ÆÆìºþ±±Ê¡Ò˲ýÊÐÒ˶¼»áËþ³ÇµØÇøºÍ²¼¿ËÈü¶ûÃɹÅ×ÔÖÎÏØÖØÇìÊÐÊÐϽÇø±±íÕÇø½¹×÷ÊÐÇßÑôÊл³ÈáÇø±¦É½ÕòÄÚÃɹźôÂ×±´¶ûÊа¢ÈÙÆìÉÂÎ÷Ê¡ÉÌÂåÊÐɽÑôÏØÉÂÎ÷Ê¡ººÖÐÊÐÁô°ÓÏظ£½¨Ê¡ÄÏƽÊÐÂí°°É½Êв©ÍûÇøÇàÐãÇø¹óÖÝÊ¡Ç­Äϲ¼ÒÀ×åÃç×å×ÔÖÎÖÝ»ÝË®ÏغÓÎ÷ÇøÌÒÔ°½ÖµÀ¸£½¨Ê¡ÄÏƽÊн¨ê±ÊÐÄÚÃɹÅÐË°²ÃËÎÚÀ¼ºÆÌØÊвýƽÇø»ØÁú¹Û½ÖµÀÇຣʡÓñÊ÷²Ø×å×ÔÖÎÖÝÖζàÏØÁÉÄþÊ¡ÌúÁëÊÐÇåºÓÇøºþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݶ÷Ê©ÊÐÄÚÃɹÅÎýÁÖ¹ùÀÕÃËÕýÏâ°×ÆìÉÂÎ÷Ê¡ººÖÐÊÐÁô°ÓÏØ»³ÈáÇøÑãÆܵØÇøº£ÄÏÊ¡ÈýɳÊÐÎ÷ɳÇø³¤°²ÇøÄÏ´åÕòÐÐÌÆÏØÁúÖÝÕòËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖÝãò¶¨ÏغÚÁú½­Ê¡´óÐË°²ÁëµØÇøºôÂêÏعóÖÝÊ¡°²Ë³ÊÐÆÕ¶¨ÏØ¿¦Ê²µØÇøÂó¸ÇÌáÏؽ­Î÷Ê¡ÒË´ºÊзîÐÂÏØÄÚÃɹÅÎýÁÖ¹ùÀÕÃËÏâ»ÆÆìËÄ´¨Ê¡¹ãÔªÊÐÄÏ¿ªÇøÌåÓýÖÐÐĽֵÀÔªÊÏÏØÔªÊÏÏØÒó´åÕòÎÚ³ľÆëÊÐɳÒÀ°Í¿ËÇøºÏ·ÊÊÐÑþº£Çø

È«Çò·þÎñÇøÓò:°¢¿ËËÕµØÇøкÍÏسç×óÊС°²ÏØÏòÑôÇø¹ÜׯµØÇøÁÉÄþÊ¡±¾ÏªÊÐϪºþÇøɽÎ÷Ê¡Ô˶¼»áÏÄÏظ£½¨Ê¡ÏÃÃÅÊкþÀïÇøɽÎ÷Ê¡ÂÀÁºÊÐá°ÏزýƽÇøÑô·»ÕòÄÚÃɹŶõ¶û¶à˹ÊжõÍпËÆìÃÜÔÆÇø¹Å±±¿ÚÕòÔªÊÏÏØ°ÙÉ«ÊÐÌïÁÖÏغþ±±Ê¡Ð¢¸ÐÊкº´¨Êй㶫ʡÉÇÍ·ÊÐÄÏ°ÄÏØɽÎ÷Ê¡½ú¶¼»á¸ßƽÊÐÇຣʡÓñÊ÷²Ø×å×ÔÖÎÖÝÖζàÏعóÖÝÊ¡°²Ë³ÊÐÆÕ¶¨ÏغÚÁú½­Ê¡´óÐË°²ÁëµØÇøºôÂêÏظÊËàÊ¡ÌìË®ÊÐÇØÖÝÇøÏòÑôÇø¾ÆÏÉÇŽֵÀÔÆÄÏÊ¡Å­½­ÀüËÛ×å×ÔÖÎÖݸ£¹±ÏØ´óÐËÇøÄÚÃɹŶõ¶û¶à˹ÊжõÍпËÆìÔÆÄÏÊ¡Àö½­ÊÐÄþÝõÒÍ×å×ÔÖÎÏؽ­Î÷ʡӥ̶ÊÐÓ཭Çø»´ÄÏÊдóͨÇøÁÉÄþÊ¡½õÖÝÊÐÒåÏØÒø´¨ÊкØÀ¼ÏØÔÞ»ÊÏØԺͷÕòÔÆÄÏÊ¡ÆÕ¶ýÊо°¶«ÒÍ×å×ÔÖÎÏØÉÂÎ÷Ê¡ººÖÐÊÐÎ÷ÏçÏØËÄ´¨Ê¡ãòÖÝÊн­ÑôÇøɽÎ÷Ê¡½úÖÐÊÐÁéʯÏؽ­ËÕÊ¡Õò½­ÊоäÈÝÊо²º£ÇøÎ÷µÔׯÕòɽÎ÷Ê¡Ë·ÖÝÊÐɽÒõÏر±³½Çøºþ±±Ê¡Ò˲ýÊÐÒÄÁêÇø¼ªÁÖÊ¡³¤´ºÊÐË«ÑôÇøÔÆÄÏÊ¡ÓñϪÊÐÐÂƽÒÍ×å´ö×å×ÔÖÎÏØɽ¶«Ê¡Ì©°²ÊÐ̩ɽÇøºÚÁú½­Ê¡´óÐË°²ÁëµØÇøºôÂêÏغþ±±Ê¡Ð¢¸ÐÊкº´¨ÊÐÄÚÃɹŶõ¶û¶à˹Êпµ°ÍʲÇø¾²º£ÇøÎ÷µÔׯÕòÉÂÎ÷Ê¡±¦¼¦ÊÐǧÑôÏØɽÎ÷Ê¡½úÖÐÊÐÌ«¹ÈÇøÎߺþÊÐÄÏÁêÏØÒø´¨ÊкØÀ¼Ïظ£½¨Ê¡ÆÎÌïÊÐÏÉÓÎÏعóÖÝÊ¡Í­ÈÊÊÐÓñÆÁ¶±×å×ÔÖÎÏØ¿¦Ê²µØÇøÂó¸ÇÌáÏØɽÎ÷Ê¡½úÖÐÊÐÁéʯÏØÔÆÄÏÊ¡Å­½­ÀüËÛ×å×ÔÖÎÖݸ£¹±Ïؽ­Î÷Ê¡¸ÓÖÝÊÐʯ³ÇÏØËÄ´¨Ê¡µÂÑôÊй㺺ÊÐÁÉÄþÊ¡ÉòÑôÊÐÉòºÓÇøÇຣʡº£±±²Ø×å×ÔÖÎÖÝÃÅÔ´»Ø×å×ÔÖÎÏØÇÅÎ÷ÇøÔ·¶«½ÖµÀº£ÄÏÊ¡ÈýɳÊÐÎ÷ɳÇøɽÎ÷Ê¡ÐÃÖÝÊж¨ÏåÏØƽ¶¥É½ÊÐÎÀ¶«ÇøɽÎ÷Ê¡½úÖÐÊÐÁéʯÏعóÖÝÊ¡°²Ë³ÊÐÆÕ¶¨ÏغϷÊÊЮÑôÇø¼»ÖÝÇø¶«Ê©¹ÅÕòËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖݾÅÁúÏØɽÎ÷Ê¡ÁÙ·ÚÊа²ÔóÏØɽ¶«Ê¡ÇൺÊгÇÑôÇøɽ¶«Ê¡ÁĶ¼»á¹ÚÏØÏòÑôÇøС¹Ø½ÖµÀºþ±±Ê¡Ð¢¸ÐÊкº´¨Êи£½¨Ê¡¸£ÖÝÊÐÓÀÌ©ÏØÁÉÄþÊ¡±¾ÏªÊÐϪºþÇøÁÉÄþÊ¡±¾ÏªÊÐϪºþÇø½­ËÕÊ¡ÐìÖÝÊÐÐÂÒÊÊÐÉÂÎ÷Ê¡Î÷°²ÊÐδÑëÇøÃÅÍ·¹µÇø´óÓø½ÖµÀÃÅÍ·¹µÇø´óÓø½ÖµÀÙñÖÝÊÐÀûÐÁÏØÔÆÄÏÊ¡ºìºÓ¹þÄá×åÒÍ×å×ÔÖÎÖÝÂÌ´ºÏØÃÜÔÆÇø¹Å±±¿ÚÕò½­ËÕÊ¡ÄÏͨÊÐÆô¶«ÊÐÇÕÖÝÊÐɽÎ÷Ê¡ÂÀÁºÊÐá°ÏØÁÉÄþÊ¡ÌúÁëÊпªÔ­Êкþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݽ¨Ê¼Ïؽ­ËÕÊ¡ËÕÖÝÊÐÏà³ÇÇøËþ³ÇµØÇøºÍ²¼¿ËÈü¶ûÃɹÅ×ÔÖÎÏØ˳ÒåÇø¿Õ¸Û½ÖµÀ²ýƽÇøСÌÀɽÕò¼»ÖÝÇø¹ÙׯÕòÂí°°É½Êв©ÍûÇø¹ã¶«Ê¡ÉîÛÚÊÐÏòÑôÇø¹ÜׯµØÇøÉÂÎ÷Ê¡ººÖÐÊÐÎ÷ÏçÏØ°ö²ºÊÐÁú×ÓºþÇøÔÆÄÏÊ¡´óÀí°××å×ÔÖÎÖÝÔÆÁúÏØËþ³ÇµØÇøºÍ²¼¿ËÈü¶ûÃɹÅ×ÔÖÎÏØ

½ñÈÕÐÐÒµ±¨¸æת´ïÕþ²ßת±äMBench: Ç廪xÌÚѶÁªºÏ½ç˵ÊÓƵÌìÏÂÄ£×ӵĺã¾ÃÓ°ÏóÄÜÁ¦£¬£¬£¬£¬ºÜÐË·ÜΪÄú½â´ðÕâ¸öÎÊÌ⣬£¬£¬£¬ÈÃÎÒÀ´°ïÄúÏêϸ˵Ã÷Ò»ÏÂ:ÊÛºó·þÎñάÐÞÖÐÐĵ绰£¬£¬£¬£¬Ö§³Ö¶àÇþµÀ·þÎñ

ÌìÏ·þÎñÇøÓò£ºÉϺ£ÊÐÊÐϽÇø¼Î¶¨Çøɽ¶«Ê¡Íþº£ÊÐËþ³ÇµØÇøºÍ²¼¿ËÈü¶ûÃɹÅ×ÔÖÎÏØɽÎ÷Ê¡ÐÃÖÝÊж¨ÏåÏغþ±±Ê¡Ò˲ýÊÐÒÄÁêÇø½­ËÕÊ¡ÄϾ©ÊÐÆÖ¿ÚÇøÉÂÎ÷Ê¡Î÷°²ÊÐδÑëÇøÄÚÃɹŶõ¶û¶à˹ÊжõÍпËÆìËÄ´¨Ê¡ãòÖÝÊн­ÑôÇøÔªÊÏÏØÒó´åÕòËÄ´¨Ê¡Á¹É½ÒÍ×å×ÔÖÎÖÝÕѾõÏظ£½¨Ê¡ÄÏƽÊн¨ê±ÊÐɽ¶«Ê¡Ì©°²Êж«Æ½ÏØËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖݹ㶫ʡ·ðɽÊÐÈýË®Çø½­ËÕÊ¡ÐìÖÝÊзáÏØƽɽÏØÕ¬±±Ïçºþ±±Ê¡ÏåÑôÊÐÏåÖÝÇø¡°²ÏØɽ¶«Ê¡µÂÖÝÊÐÆëºÓÏØɽÎ÷Ê¡Ô˶¼»áÏÄÏظ£½¨Ê¡ÏÃÃÅÊк£²×ÇøÉÂÎ÷Ê¡±¦¼¦ÊÐüÏØÄÚÃɹÅÎýÁÖ¹ùÀÕÃËÏâ»ÆÆìÎ÷²ØɽÄÏÊÐɽÎ÷Ê¡Ô˶¼»áÏÄÏغÚÁú½­Ê¡ÒÁ´ºÊнðÁÖÇø¾®ÚêÏزâÓãÕòÏòÑôÇøС¹Ø½ÖµÀɽÎ÷Ê¡½úÖÐÊÐÁéʯÏØÔÞ»ÊÏØÎ÷ÁúÃÅÏçËÄ´¨Ê¡¸Ê×βØ×å×ÔÖÎÖݾÅÁúÏؼ»ÖÝÇø¹ÙׯÕò¸£½¨Ê¡ÏÃÃÅÊк£²×ÇøÉÂÎ÷Ê¡±¦¼¦ÊÐǧÑôÏغÚÁú½­Ê¡¼Ñľ˹ÊÐÌÀÔ­ÏØÃÜÔÆÇø¸ßÁëÕò½­ËÕÊ¡ÄϾ©ÊÐÆÖ¿ÚÇøÃÅÍ·¹µÇø´óÓø½ÖµÀÉÂÎ÷Ê¡ººÖÐÊÐÎ÷ÏçÏغþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݶ÷Ê©ÊÐÎ÷²ØɽÄÏÊй±¸ÂÏØÎ÷ÇàÇø¾«ÎäÕò¹óÖÝÊ¡ÁùÅÌË®ÊÐË®³ÇÏØÂí°°É½Êв©ÍûÇøÒÁÀç¹þÈø¿Ë×ÔÖÎÖÝÕÑËÕÏØÏòÑôÇø¹ÜׯµØÇø½­Î÷Ê¡ÒË´ºÊÐÒË·áÏØɽÎ÷Ê¡½úÖÐÊÐÌ«¹ÈÇøÔÆÄÏÊ¡ºìºÓ¹þÄá×åÒÍ×å×ÔÖÎÖÝãòÎ÷Ïؽ­ËÕÊ¡ÐìÖÝÊзáÏزýƽÇøÑô·»Õòºþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݶ÷Ê©Êкþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݽ¨Ê¼ÏØ°ö²ºÊÐÁú×ÓºþÇøÎ÷²ØɽÄÏÊй±¸ÂÏØ¡°²ÏØÁÉÄþÊ¡½õÖÝÊÐÒåÏØƽ¶¥É½ÊÐÕ¿ºÓÇø±±³½Çø¡°²ÏØ°ÍÒô¹ùÀãÃɹÅ×ÔÖÎÖݺÍ˶Ïؽ­ËÕÊ¡ÐìÖÝÊÐÉÂÎ÷Ê¡ººÖÐÊÐÁô°ÓÏع㶫ʡ·ðɽÊÐÈýË®ÇøƽɽÏض«»ØÉáÕò¸ÊËàʡ¤ÄÏÊÐÎ䶼ÇøËÄ´¨Ê¡Ò˱öÊÐçîÏر±³½ÇøÁÉÄþÊ¡½õÖÝÊÐÒåÏؼªÁÖÊ¡ËÄƽÊÐÌúÎ÷Çø²ýƽÇøСÌÀɽÕòÙñÖÝÊÐÀûÐÁÏع㶫ʡÉÇÍ·ÊÐÄÏ°ÄÏØɽ¶«Ê¡ÁĶ¼»áÜÝƽÇø½­Î÷Ê¡¼ª°²ÊÐÓÀÐÂÏØɽ¶«Ê¡×Ͳ©ÊÐÁÙ×ÍÇøºþ±±Ê¡¶÷Ê©ÍÁ¼Ò×åÃç×å×ÔÖÎÖݶ÷Ê©Êн­Î÷Ê¡¾Å½­ÊÐÎäÄþÏØɽ¶«Ê¡×Ͳ©ÊÐÁÙ×ÍÇøËÄ´¨Ê¡¹ãÔªÊй㶫ʡ¹ãÖÝÊÐÔ½ÐãÇø·ą́ÇøºÍÒå½ÖµÀºþ±±Ê¡Ð¢¸ÐÊкº´¨ÊÐå§ÑôÊÐÄÏÀÖÏØÁÉÄþÊ¡±¾ÏªÊÐϪºþÇøÔªÊÏÏØËÕ´åÏçÔÆÄÏÊ¡ÎÄɽ׳×åÃç×å×ÔÖÎÖݹãÄÏÏØƽɽÏظÚÄÏÕò½­Î÷Ê¡¼ª°²Êмª°²ÏØɽ¶«Ê¡Î«·»ÊÐÇàÖÝÊÐÎ÷²ØÀ­ÈøÊдï×ÎÇø¸£½¨Ê¡ÄÏƽÊйÌÔ­ÊÐÎ÷¼ªÏØÄÏ¿ªÇø³¤ºç½ÖµÀ¸£½¨Ê¡¸£ÖÝÊÐÓÀÌ©ÏØËÄ´¨Ê¡µÂÑôÊй㺺Êн­Î÷Ê¡¾°µÂÕòÊвý½­Çø½¹×÷ÊÐÖÐÕ¾ÇøÁÉÄþÊ¡ÌúÁëÊпªÔ­ÊÐ

ÊÛºó·þÎñÉÏÃÅ·þÎñµç»°£¬£¬£¬£¬ÖÇÄÜ·ÖÅäƱ¾Ý£ºMBench: Ç廪xÌÚѶÁªºÏ½ç˵ÊÓƵÌìÏÂÄ£×ӵĺã¾ÃÓ°ÏóÄÜÁ¦

Ëæ×ÅÊÓƵÌìÍâÐÐÒÕµÄÉú³¤£¬£¬£¬£¬Ä£×ÓÕýÔÚ´Ó¶ÌÊÓƵƬ¶ÏºÏ³É£¬£¬£¬£¬ÏòÁ÷ʽ³¤ÊÓƵÌìÉúÑݽø¡£¡£È»¶ø£¬£¬£¬£¬½ö½ö×öµ½ÊÓ¾õÉϵıÆÕæÊDz»·óµÄ¡£¡£Ò»¸ö¹¦Ð§ÍêÕûµÄÊÓƵÌìÏÂÄ£×Ó£¬£¬£¬£¬±ØÐèÄܹ»ÔÚ³¤Ê±Ðò½»»¥Öмá³ÖÎȹ̵ÄÄÚ²¿×´Ì¬£¬£¬£¬£¬²¢×ñÕÕÕæʵÌìϵÄÎïÀí¶¨ÂÉÓëÂß¼­¹æÔò¡£¡£

ΪÁËϵͳ»¯µØÁ¿»¯ºÍÆÀ¹ÀÕâÒ»½¹µãÄÜÁ¦£¬£¬£¬£¬Ç廪´óѧÓëÌÚѶ΢ÐÅÊÓ¾õÍŶÓÍƳöÁËMBench¡ª¡ª Ò»¸öרÃÅÕë¶ÔÁ÷ʽÊÓƵÌìÉúÄ£×ÓºÍÌìÏÂÄ£×ÓÓ°ÏóÄÜÁ¦µÄÆÀ²â»ù×¼¡£¡£

ÏîÄ¿Ö÷Ò³£º https://peanutup.github.io/MBench-project/´úÂë¿ÍÕ»£º https://github.com/study-overflow/MBenchÅÅÐаñÁ´½Ó£ºhttps://huggingface.co/spaces/study-overflow/MBench_Leaderboard

Ϊʲô¹Ø×¢ ¡°Ó°ÏóÄÜÁ¦¡±£¿£¿£¿£¿£¿£¿

ËäÈ»ÏÖÓеÄÊÓƵÌìÍâÐÐÒÕÒѾ­¿ÉÒԺϳÉÊÓ¾õÉÏ¿ÉÐŵÄÊÓƵƬ¶Ï£¬£¬£¬£¬µ«ÔÚ×ßÏò³¤ÊÓƵÌìÉúºÍ¹¦Ð§ÐÔµÄÌìÏÂÄ£Äâʱ£¬£¬£¬£¬Ò»¸ö¸ùÌìÐÔµÄÌôÕ½ÒÀÈ»±£´æ£ºÄ£×Ó±ØÐèÄܹ»ÖÒʵµØά³ÖÏÖʵÌìϵÄÊÓ¾õÌØÕ÷¡¢ÓïÒå¹æÔò¡¢ÄÚÔÚ¶¯Á¦Ñ§ºÍÎïÀí¼ÍÂÉ£¬£¬£¬£¬²¢¿É¿¿µØÖ§³Ö³¤Ê±ÐòµÄÕ¹Íû¡¢ÍÆÀíÓë½»»¥¡£¡£ÈôÊÇȱ·¦ÕâÖÖÄÜÁ¦£¬£¬£¬£¬Ëæ×ÅÌìÉúʱ¼äµÄÑÓÉì»ò½»»¥µÄÔöÌí£¬£¬£¬£¬ÊÓƵÍùÍù»á·ºÆðÎïÌåºÍÈËÎïÊÓ¾õÌØÕ÷Í»±ä¡¢³¡¾°½á¹¹±ÀËú»òÒò¹û¹Øϵ¹ýʧµÈÕ÷Ï󡣡£

ÕâÖÖ´Ó ¡°ÊÓ¾õºÏ³É¡± Ïò ¡°ÌìϽ¨Ä£¡± µÄ¿çÔ½£¬£¬£¬£¬ÊµÖÊÉÏÐèҪģ×Ӿ߱¸ºã¾ÃÓ°ÏóÄÜÁ¦£¬£¬£¬£¬ÒÔά³ÖÄÚ²¿ÌìÏÂ״̬ÔÚ³¤Ê±ÐòºÍÖØ´ó½»»¥ÖеÄÎȹÌÐÔÓëÒ»ÖÂÐÔ¡£¡£

È»¶ø£¬£¬£¬£¬Ä¿½ñµÄÊÓƵÌìÉúÆÀ²â»ù×¼£¨Èç VBench µÈ£©Ö÷Òª×ÅÖØÓÚµ¥´Î¶ÌÊÓƵÌìÉúµÄÊÓ¾õÖÊÁ¿¡¢Ô˶¯Á¬¹áÐÔ»òÎı¾¶ÔÆë¡£¡£ÕâЩָ±êÄܹ»È¨ºâÌìÉúµÄÊÓƵ ¡°¿´ÆðÀ´Õæ²»Õ桱£¬£¬£¬£¬È´ºöÂÔÁ˶ÔÌìÏ ¡°³¤ÆÚÊôÐÔ¡± ±£´æÄÜÁ¦µÄÁ¿»¯¡£¡£ÎªÁËÌî²¹ÕâÒ»¿Õȱ£¬£¬£¬£¬ÏµÍ³ÐÔµØȨºâÄ£×ÓÄÜ·ñÕæÕý¼Ç×Ų¢Ä£ÄâÎïÀíÌìÏ£¬£¬£¬£¬MBench Ê״ν«ÊÓƵºÍÌìÏÂÄ£×ÓÔÚ³¤Ê±ÐòϵÄÓ°ÏóÄÜÁ¦×÷Ϊ½¹µãÊӲ칤¾ß¾ÙÐÐÁËרÏî»ù×¼²âÊÔ¡£¡£

MBench ºÍÖ÷Á÷ÊÓƵÌìÏÂÄ£×Ó benchmark µÄ½ÏÁ¿

MBench µÄ¶àά¶È¼Ü¹¹

MBench »ùÓÚ 1040 ¸ö°¸Àý£¬£¬£¬£¬½«Ó°ÏóÄÜÁ¦½â¹¹ÎªÈý¸ö»¥²¹µÄ½¹µãά¶È£¬£¬£¬£¬²¢½øÒ»²½Ï¸·ÖΪ 12 ¸ö¿ÉÁ¿»¯µÄ×Óά¶È£¬£¬£¬£¬º­¸ÇÁË´Ó¾²Ì¬ÊôÐÔµ½¶¯Ì¬Òò¹ûµÄÈ«·½Î»Ó°ÏóÒªÇó£º

MBench µÄÆÀ²âά¶È»®·Ö

ʵÌåÒ»ÖÂÐÔ (Entity Consistency)£º

ʵÌåÒ»ÖÂÐÔ¹Øעģ×Ó±£´æ¸öÌå¼ÓÈëÕß³¤ÆÚÉí·ÝºÍÊôÐÔµÄÄÜÁ¦¡£¡£°üÀ¨ÎïÌåÒ»ÖÂÐÔ£¨¼¸ºÎ¡¢ÎÆÀí£©ºÍÈËÀàÒ»ÖÂÐÔ£¨Éí·ÝÌØÕ÷¡¢Íâ¹Ûϸ½Ú£©¡£¡£¼òÆÓÀ´Ëµ£¬£¬£¬£¬¾ÍÊǵ±Ò»¸öÈË»òÎïÌå±»ÕÚµ²¡¢ÍÑÀë»­ÃæÔÙ»ØÀ´Ê±£¬£¬£¬£¬ËüµÄÑù×Ó¡¢ÒÂ×Å¡¢ÌØÕ÷ÊÇ·ñ¼á³ÖÎȹÌ¡£¡£

ÇéÐÎÒ»ÖÂÐÔ (Environment Consistency)£º

ÇéÐÎÒ»ÖÂÐÔȨºâÄ£Äⳡ¾°µÄÎȹÌÐÔ¡£¡£°üÀ¨¿Õ¼äÒ»ÖÂÐÔ£¨Í¨¹ý¼«Ïß¼¸ºÎºÍÖØͶӰÎó²îȨºâ 3D ½á¹¹£©ºÍäÖȾһÖÂÐÔ£¨¹âÕÕÓëÆø¸ÅµÄÎȹÌÐÔ£©¡£¡£»£»£»£ÇéÐÎÒ»ÖÂÐÔÄ¥Á·µÄÊÇÄ£×Ó¶Ô 3D ¿Õ¼äÌìϵÄÃ÷È·ÄÜÁ¦ ¡ª¡ª µ±Ïà»úÒƶ¯¡¢ÐýתÔٻص½Ô­Î»Ê±£¬£¬£¬£¬·¿¼äµÄ½á¹¹¡¢³¡¾°µÄ½á¹¹ÊÇ·ñºÍ֮ǰ¼á³ÖÒ»Ö£»£»£»£»¶øäÖȾһÖÂÐÔȨºâÊÓƵÕûÌåÉϹâÏßµÄÆ«ÏòºÍÕûÌåµÄ»­ÃæÆø¸ÅÊÇ·ñÄܼá³ÖÒ»Ö¡£¡£

Òò¹ûÒ»ÖÂÐÔ (Causal Consistency)£º

Òò¹ûÒ»ÖÂÐÔȨºâÄ£×ÓÄÜ·ñ¼Ç×ÅÊÂÎñµÄÒò¹ûÂß¼­£¬£¬£¬£¬·ÖΪ×ÔÑÝ»¯ºÍ½»»¥¡£¡£ÕâÊÇ×î¸ß¼¶µÄÓ°ÏóÄÜÁ¦£ºÀýÈ磬£¬£¬£¬µ±Ò»¸öÎïÌåÕýÔÚ±»´òË飬£¬£¬£¬¶øÏà»úÒÆ¿ªÔÙת»Ø£¬£¬£¬£¬µØÉÏÓ¦¸Ã±¬·¢ÏìÓ¦µÄËéƬ£»£»£»£»µ±Äãͨ¹ýÎı¾Ö¸ÁîÄ£×ÓÒƶ¯Ò»¸öÎïÌ壬£¬£¬£¬Ä£×ÓÓ¦¸Ã׼ȷִÐв¢¼Ç×ÅеÄλÖ㬣¬£¬£¬×ÝÈ»ÎïÌåÔÚÄ¿½ñµÄ»­ÃæÖÐÒѾ­±äµÃ²»¿É¼û¡£¡£

Õë¶ÔÌìÉúËæ»úÐÔµÄÆÀ·Ö»úÖÆ£ºTrigger-Conditioned Scoring

ÔÚÆÀ¹ÀÊÓƵģ×ÓµÄÓ°ÏóÄÜÁ¦Ê±£¬£¬£¬£¬ÍŶӷ¢Ã÷ÁËÒ»¸öÖ÷ÒªµÄ×ÌÈÅÒòËØ£¨confound£©¡ª¡ª Ä£×Ó¶ÔÓ°Ïó´¥·¢ÊÂÎñµÄÏìÓ¦ÄÜÁ¦±£´æ²î±ð¡£¡£ÀýÈ磬£¬£¬£¬¹ØÓÚÎı¾Ìõ¼þÄ£×Ó£¬£¬£¬£¬ÆäÌìÉúµÄÊÓƵÍùÍùÎÞ·¨ÕæÕý·ºÆðÌáÐÑ´ÊÖÐÒªÇóµÄ¾µÍ·Ô˶¯¡¢ÎïÌåÕÚµ²»ò¶¯Ì¬×ª±ä£»£»£»£»¹ØÓÚÐж¯Çý¶¯µÄÌìÏÂÄ£×Ó£¬£¬£¬£¬Ò²¿ÉÄܱ£´æÌìÉúµÄÊÓƵÎÞ·¨×¼È·ÏìÓ¦Ïà»úÔ˶¯µÄÕ÷Ï󡣡£Õâµ¼ÖÂÄ£×Ó¿ÉÄÜ»áͨ¹ýÌìÉú¾²Ì¬»òÊؾÉÄÚÈݵķ½Ê½À´±Ü¿ªÌôÕ½£¬£¬£¬£¬´Ó¶ø»ñµÃÐé¸ßµÄÒ»ÖÂÐÔÆÀ·Ö¡£¡£

Ϊ´Ë£¬£¬£¬£¬MBench ÒýÈëÁË´¥·¢Ìõ¼þÆÀ·Ö£¨Trigger-Conditioned Scoring£©»úÖÆ£¬£¬£¬£¬½«µÃ·Ö²ð½âΪÁ½¸ö²¿·Ö£º

1.´¥·¢ÁýÕÖÂÊ (Trigger Coverage, C_trig)£ºÑé֤ģ×ÓÊÇ·ñÀÖ³ÉÖ´ÐÐÁËÓ°ÏóÌôÕ½ÊÂÎñ£¨ÈçÎïÌå³ö¾µÔÙÈë¾µ£©¡£¡£

2.Ó°Ïó¿É¿¿¶È (Memory Reliability, S_rel)£º½öÔÚÀֳɴ¥·¢ÌôÕ½µÄÑù±¾ÉÏÅÌËãÒ»ÖÂÐԵ÷Ö¡£¡£

×îÖÕµÄ M-Score È¡Á½Õߵĵ÷Çå¾²¾ùÊý£¬£¬£¬£¬Ö¼ÔÚ´¦·Öͨ¹ýÌìÉúÊØ¾É / ¾²Ì¬ÄÚÈÝÀ´¹æ±ÜÒ»ÖÂÐÔÌôÕ½µÄÐÐΪ£¬£¬£¬£¬½±ÀøÄÇЩ¼ÈÄÜÄ£Ä⶯̬ÌìÏÂÓÖÄܼá³ÖÒ»ÖÂÐÔµÄÄ£×Ó¡£¡£

¶Ô 14 ¸öÖ÷Á÷ SOTA Ä£×ÓµÄÆÀ²â·¢Ã÷

MBench ¶Ô 8 ¸öÎı¾Çý¶¯Ä£×ÓºÍ 6 ¸öÐж¯Çý¶¯Ä£×Ó¾ÙÐÐÁË´ó¹æÄ£ÆÀ²â£¬£¬£¬£¬ÆÀ²âЧ¹ûÅú×¢£¬£¬£¬£¬ÏÖÔÚ²¢²»±£´æ¼òµ¥Ä£×Ó¿ÉÒÔÔÚËùÓÐά¶ÈÉ϶¼ÌåÏÖ¾«²Ê£¬£¬£¬£¬Ó°ÏóÄÜÁ¦ÈÔÈ»ÊÇÁ÷ʽÊÓƵÌìÉúºÍÌìÏÂÄ£×ÓµÄÆÕ±éÆ¿¾±¡£¡£

Ä£×Ó·ÖÊýÀ×´ïͼ

ʵÑéÆÀ²âЧ¹û

¶ÔʵÑéЧ¹û¾ÙÐÐÆÊÎö£¬£¬£¬£¬»¹¿ÉÒÔ»ñµÃÈçÏ·¢Ã÷£º

¿Õ¼äÓëÒò¹ûÄÜÁ¦ÊÇÖ÷Ҫƿ¾±£ºÆÀ²âЧ¹ûÏÔʾ£¬£¬£¬£¬´ó¶¼Ä£×ÓÔÚ³¤Ê±ÐòÊÓ½Çת»»ÏµĿռ伸ºÎ»¹Ô­£¬£¬£¬£¬ÒÔ¼°Éæ¼°ÎïÀíÑÝ»¯µÄÒò¹ûÂß¼­¼á³ÖÉϱ£´æÏÔÖøȱÏÝ£¬£¬£¬£¬ÄÑÒÔÕÒ»ØÏûÊŵÄÊÓͼ»òÑÓÐø»­ÃæÍâµÄÎïÀíÀú³Ì¡£¡£Ðж¯Çý¶¯Ä£× ¡°Æ«¿Æ¡± Õ÷Ïó£ºÐж¯Ìõ¼þÄ£×ÓËäÈ»ÔÚ¿Õ¼äÎȹÌÐÔÖ¸±êÉÏÌåÏֽϺ㬣¬£¬£¬µ«±£´æÒ»¸öÆÕ±éµÄʧЧģʽ ¡ª¡ª ¼´ÇãÏòÓÚÌìÉúÌ«¹ý¾²Ì¬µÄ³¡¾°¡£¡£ÕâÖÖ·½Ê½ËäÈ»¹æ±ÜÁË¿Õ¼äÌ®Ëõ£¬£¬£¬£¬µ«Ä£×ÓÏÖʵÉÏÎÞ·¨Çý¶¯ÖØ´óµÄÎïÀíÑÝ»¯¡£¡£ÊÓ¾õ±ÆÕæ¶È²»¼´ÊÇÓ°ÏóÎȹÌÐÔ£ºÊµÑé֤ʵ£¬£¬£¬£¬Äܹ»ºÏ³É¸ßÖÊÁ¿¡¢¸ß±£Õæ»­ÃæµÄÄ£×Ó£¬£¬£¬£¬ÔÚ³¤Ê±ÐòÓ°Ïóά¶ÈÉÏ·×Æ綨ռÓÅ¡£¡£Õâ˵Ã÷´¿´âµÄÊÓ¾õÌìÉúÖ¸±êÎÞ·¨Ìæ»»¶ÔÄ£×ÓÄÚ²¿×´Ì¬³¤ÆÚÐÔµÄרÏîÆÀ¹À£¬£¬£¬£¬Ö¤ÊµÎú½¨ÉèÓ°ÏóÄÜÁ¦»ù×¼µÄÐëÒªÐÔ¡£¡£

´ÓÌìÉúµ¥ÕÅͼƬµ½ºÏÒòËØÖÓ¼¶ÊÓƵ£¬£¬£¬£¬ÊÓƵÌìÍâÐÐÒÕÒѾ­È¡µÃÁËÁîÈËÖõÄ¿µÄÇ°½ø¡£¡£È»¶ø£¬£¬£¬£¬ÒªÊµÏÖÄܹ»Ã÷È·¡¢Õ¹Íû²¢¾ÙÐн»»¥µÄÌìÏÂÄ£×Ó£¬£¬£¬£¬²¢ÈÃÁ÷ʽÊÓƵÌìÉú¼á³Ö³¤Ê±Ò»ÖÂÐÔ£¬£¬£¬£¬ÎÒÃÇÈÔÓкܳ¤µÄ·Ҫ×ß¡£¡£¶ø ¡°Ó°Ïó¡±£¬£¬£¬£¬ÕýÊǹ¹½¨ÕâЩ½¹µãÄÜÁ¦µÄ»ùʯ¡£¡£MBench Õ¹ÏÖÁËÄ¿½ñÖ÷Á÷Ä£×ÓÔÚÓ°ÏóÄÜÁ¦ÉϵÄÕæʵ½çÏߣ¬£¬£¬£¬Ò²ÎªÎ´À´µÄÑо¿Ö¸Ã÷ÎúÆ«Ïò¡£¡£

ÏÖÔÚ£¬£¬£¬£¬MBench ÒÑÖÜÈ«¿ªÔ´£¬£¬£¬£¬¿ªÔ´ÄÚÈÝ°üÀ¨£ºÍêÕûµÄ 1040 ¸öÆÀ²â°¸ÀýÊý¾Ý¼¯¡¢×Ô¶¯»¯ÆÀ²â´úÂëÓ빤¾ßÁ´¡¢ÊµÊ±¸üеĹûÕæÅÅÐаñ£¬£¬£¬£¬ÒÔ¼°ÏêϸµÄÊÖÒÕ±¨¸æÓëʵÑéЧ¹û¡£¡£ÏàÐÅÔÚ MBench µÄÍƶ¯Ï£¬£¬£¬£¬ÎÒÃÇÖÕ½«Ó­À´Äܹ» ¡°¼Ç×ÅÌìÏ¡¢Ã÷È·ÌìÏ¡¢Õ¹ÍûÌìÏ¡± µÄÏÂÒ»´úÊÓƵÌìÏÂÄ£×Ó¡£¡£

±¾ÔÂÏà¹Ø²¿·ÖÐû²¼Ö÷Òª±¨¸æMBench: Ç廪xÌÚѶÁªºÏ½ç˵ÊÓƵÌìÏÂÄ£×ӵĺã¾ÃÓ°ÏóÄÜÁ¦

Ëæ×ÅÊÓƵÌìÍâÐÐÒÕµÄÉú³¤£¬£¬£¬£¬Ä£×ÓÕýÔÚ´Ó¶ÌÊÓƵƬ¶ÏºÏ³É£¬£¬£¬£¬ÏòÁ÷ʽ³¤ÊÓƵÌìÉúÑݽø¡£¡£È»¶ø£¬£¬£¬£¬½ö½ö×öµ½ÊÓ¾õÉϵıÆÕæÊDz»·óµÄ¡£¡£Ò»¸ö¹¦Ð§ÍêÕûµÄÊÓƵÌìÏÂÄ£×Ó£¬£¬£¬£¬±ØÐèÄܹ»ÔÚ³¤Ê±Ðò½»»¥Öмá³ÖÎȹ̵ÄÄÚ²¿×´Ì¬£¬£¬£¬£¬²¢×ñÕÕÕæʵÌìϵÄÎïÀí¶¨ÂÉÓëÂß¼­¹æÔò¡£¡£

ΪÁËϵͳ»¯µØÁ¿»¯ºÍÆÀ¹ÀÕâÒ»½¹µãÄÜÁ¦£¬£¬£¬£¬Ç廪´óѧÓëÌÚѶ΢ÐÅÊÓ¾õÍŶÓÍƳöÁËMBench¡ª¡ª Ò»¸öרÃÅÕë¶ÔÁ÷ʽÊÓƵÌìÉúÄ£×ÓºÍÌìÏÂÄ£×ÓÓ°ÏóÄÜÁ¦µÄÆÀ²â»ù×¼¡£¡£

ÏîÄ¿Ö÷Ò³£º https://peanutup.github.io/MBench-project/´úÂë¿ÍÕ»£º https://github.com/study-overflow/MBenchÅÅÐаñÁ´½Ó£ºhttps://huggingface.co/spaces/study-overflow/MBench_Leaderboard

Ϊʲô¹Ø×¢ ¡°Ó°ÏóÄÜÁ¦¡±£¿£¿£¿£¿£¿£¿

ËäÈ»ÏÖÓеÄÊÓƵÌìÍâÐÐÒÕÒѾ­¿ÉÒԺϳÉÊÓ¾õÉÏ¿ÉÐŵÄÊÓƵƬ¶Ï£¬£¬£¬£¬µ«ÔÚ×ßÏò³¤ÊÓƵÌìÉúºÍ¹¦Ð§ÐÔµÄÌìÏÂÄ£Äâʱ£¬£¬£¬£¬Ò»¸ö¸ùÌìÐÔµÄÌôÕ½ÒÀÈ»±£´æ£ºÄ£×Ó±ØÐèÄܹ»ÖÒʵµØά³ÖÏÖʵÌìϵÄÊÓ¾õÌØÕ÷¡¢ÓïÒå¹æÔò¡¢ÄÚÔÚ¶¯Á¦Ñ§ºÍÎïÀí¼ÍÂÉ£¬£¬£¬£¬²¢¿É¿¿µØÖ§³Ö³¤Ê±ÐòµÄÕ¹Íû¡¢ÍÆÀíÓë½»»¥¡£¡£ÈôÊÇȱ·¦ÕâÖÖÄÜÁ¦£¬£¬£¬£¬Ëæ×ÅÌìÉúʱ¼äµÄÑÓÉì»ò½»»¥µÄÔöÌí£¬£¬£¬£¬ÊÓƵÍùÍù»á·ºÆðÎïÌåºÍÈËÎïÊÓ¾õÌØÕ÷Í»±ä¡¢³¡¾°½á¹¹±ÀËú»òÒò¹û¹Øϵ¹ýʧµÈÕ÷Ï󡣡£

ÕâÖÖ´Ó ¡°ÊÓ¾õºÏ³É¡± Ïò ¡°ÌìϽ¨Ä£¡± µÄ¿çÔ½£¬£¬£¬£¬ÊµÖÊÉÏÐèҪģ×Ӿ߱¸ºã¾ÃÓ°ÏóÄÜÁ¦£¬£¬£¬£¬ÒÔά³ÖÄÚ²¿ÌìÏÂ״̬ÔÚ³¤Ê±ÐòºÍÖØ´ó½»»¥ÖеÄÎȹÌÐÔÓëÒ»ÖÂÐÔ¡£¡£

È»¶ø£¬£¬£¬£¬Ä¿½ñµÄÊÓƵÌìÉúÆÀ²â»ù×¼£¨Èç VBench µÈ£©Ö÷Òª×ÅÖØÓÚµ¥´Î¶ÌÊÓƵÌìÉúµÄÊÓ¾õÖÊÁ¿¡¢Ô˶¯Á¬¹áÐÔ»òÎı¾¶ÔÆë¡£¡£ÕâЩָ±êÄܹ»È¨ºâÌìÉúµÄÊÓƵ ¡°¿´ÆðÀ´Õæ²»Õ桱£¬£¬£¬£¬È´ºöÂÔÁ˶ÔÌìÏ ¡°³¤ÆÚÊôÐÔ¡± ±£´æÄÜÁ¦µÄÁ¿»¯¡£¡£ÎªÁËÌî²¹ÕâÒ»¿Õȱ£¬£¬£¬£¬ÏµÍ³ÐÔµØȨºâÄ£×ÓÄÜ·ñÕæÕý¼Ç×Ų¢Ä£ÄâÎïÀíÌìÏ£¬£¬£¬£¬MBench Ê״ν«ÊÓƵºÍÌìÏÂÄ£×ÓÔÚ³¤Ê±ÐòϵÄÓ°ÏóÄÜÁ¦×÷Ϊ½¹µãÊӲ칤¾ß¾ÙÐÐÁËרÏî»ù×¼²âÊÔ¡£¡£

MBench ºÍÖ÷Á÷ÊÓƵÌìÏÂÄ£×Ó benchmark µÄ½ÏÁ¿

MBench µÄ¶àά¶È¼Ü¹¹

MBench »ùÓÚ 1040 ¸ö°¸Àý£¬£¬£¬£¬½«Ó°ÏóÄÜÁ¦½â¹¹ÎªÈý¸ö»¥²¹µÄ½¹µãά¶È£¬£¬£¬£¬²¢½øÒ»²½Ï¸·ÖΪ 12 ¸ö¿ÉÁ¿»¯µÄ×Óά¶È£¬£¬£¬£¬º­¸ÇÁË´Ó¾²Ì¬ÊôÐÔµ½¶¯Ì¬Òò¹ûµÄÈ«·½Î»Ó°ÏóÒªÇó£º

MBench µÄÆÀ²âά¶È»®·Ö

ʵÌåÒ»ÖÂÐÔ (Entity Consistency)£º

ʵÌåÒ»ÖÂÐÔ¹Øעģ×Ó±£´æ¸öÌå¼ÓÈëÕß³¤ÆÚÉí·ÝºÍÊôÐÔµÄÄÜÁ¦¡£¡£°üÀ¨ÎïÌåÒ»ÖÂÐÔ£¨¼¸ºÎ¡¢ÎÆÀí£©ºÍÈËÀàÒ»ÖÂÐÔ£¨Éí·ÝÌØÕ÷¡¢Íâ¹Ûϸ½Ú£©¡£¡£¼òÆÓÀ´Ëµ£¬£¬£¬£¬¾ÍÊǵ±Ò»¸öÈË»òÎïÌå±»ÕÚµ²¡¢ÍÑÀë»­ÃæÔÙ»ØÀ´Ê±£¬£¬£¬£¬ËüµÄÑù×Ó¡¢ÒÂ×Å¡¢ÌØÕ÷ÊÇ·ñ¼á³ÖÎȹÌ¡£¡£

ÇéÐÎÒ»ÖÂÐÔ (Environment Consistency)£º

ÇéÐÎÒ»ÖÂÐÔȨºâÄ£Äⳡ¾°µÄÎȹÌÐÔ¡£¡£°üÀ¨¿Õ¼äÒ»ÖÂÐÔ£¨Í¨¹ý¼«Ïß¼¸ºÎºÍÖØͶӰÎó²îȨºâ 3D ½á¹¹£©ºÍäÖȾһÖÂÐÔ£¨¹âÕÕÓëÆø¸ÅµÄÎȹÌÐÔ£©¡£¡£»£»£»£ÇéÐÎÒ»ÖÂÐÔÄ¥Á·µÄÊÇÄ£×Ó¶Ô 3D ¿Õ¼äÌìϵÄÃ÷È·ÄÜÁ¦ ¡ª¡ª µ±Ïà»úÒƶ¯¡¢ÐýתÔٻص½Ô­Î»Ê±£¬£¬£¬£¬·¿¼äµÄ½á¹¹¡¢³¡¾°µÄ½á¹¹ÊÇ·ñºÍ֮ǰ¼á³ÖÒ»Ö£»£»£»£»¶øäÖȾһÖÂÐÔȨºâÊÓƵÕûÌåÉϹâÏßµÄÆ«ÏòºÍÕûÌåµÄ»­ÃæÆø¸ÅÊÇ·ñÄܼá³ÖÒ»Ö¡£¡£

Òò¹ûÒ»ÖÂÐÔ (Causal Consistency)£º

Òò¹ûÒ»ÖÂÐÔȨºâÄ£×ÓÄÜ·ñ¼Ç×ÅÊÂÎñµÄÒò¹ûÂß¼­£¬£¬£¬£¬·ÖΪ×ÔÑÝ»¯ºÍ½»»¥¡£¡£ÕâÊÇ×î¸ß¼¶µÄÓ°ÏóÄÜÁ¦£ºÀýÈ磬£¬£¬£¬µ±Ò»¸öÎïÌåÕýÔÚ±»´òË飬£¬£¬£¬¶øÏà»úÒÆ¿ªÔÙת»Ø£¬£¬£¬£¬µØÉÏÓ¦¸Ã±¬·¢ÏìÓ¦µÄËéƬ£»£»£»£»µ±Äãͨ¹ýÎı¾Ö¸ÁîÄ£×ÓÒƶ¯Ò»¸öÎïÌ壬£¬£¬£¬Ä£×ÓÓ¦¸Ã׼ȷִÐв¢¼Ç×ÅеÄλÖ㬣¬£¬£¬×ÝÈ»ÎïÌåÔÚÄ¿½ñµÄ»­ÃæÖÐÒѾ­±äµÃ²»¿É¼û¡£¡£

Õë¶ÔÌìÉúËæ»úÐÔµÄÆÀ·Ö»úÖÆ£ºTrigger-Conditioned Scoring

ÔÚÆÀ¹ÀÊÓƵģ×ÓµÄÓ°ÏóÄÜÁ¦Ê±£¬£¬£¬£¬ÍŶӷ¢Ã÷ÁËÒ»¸öÖ÷ÒªµÄ×ÌÈÅÒòËØ£¨confound£©¡ª¡ª Ä£×Ó¶ÔÓ°Ïó´¥·¢ÊÂÎñµÄÏìÓ¦ÄÜÁ¦±£´æ²î±ð¡£¡£ÀýÈ磬£¬£¬£¬¹ØÓÚÎı¾Ìõ¼þÄ£×Ó£¬£¬£¬£¬ÆäÌìÉúµÄÊÓƵÍùÍùÎÞ·¨ÕæÕý·ºÆðÌáÐÑ´ÊÖÐÒªÇóµÄ¾µÍ·Ô˶¯¡¢ÎïÌåÕÚµ²»ò¶¯Ì¬×ª±ä£»£»£»£»¹ØÓÚÐж¯Çý¶¯µÄÌìÏÂÄ£×Ó£¬£¬£¬£¬Ò²¿ÉÄܱ£´æÌìÉúµÄÊÓƵÎÞ·¨×¼È·ÏìÓ¦Ïà»úÔ˶¯µÄÕ÷Ï󡣡£Õâµ¼ÖÂÄ£×Ó¿ÉÄÜ»áͨ¹ýÌìÉú¾²Ì¬»òÊؾÉÄÚÈݵķ½Ê½À´±Ü¿ªÌôÕ½£¬£¬£¬£¬´Ó¶ø»ñµÃÐé¸ßµÄÒ»ÖÂÐÔÆÀ·Ö¡£¡£

Ϊ´Ë£¬£¬£¬£¬MBench ÒýÈëÁË´¥·¢Ìõ¼þÆÀ·Ö£¨Trigger-Conditioned Scoring£©»úÖÆ£¬£¬£¬£¬½«µÃ·Ö²ð½âΪÁ½¸ö²¿·Ö£º

1.´¥·¢ÁýÕÖÂÊ (Trigger Coverage, C_trig)£ºÑé֤ģ×ÓÊÇ·ñÀÖ³ÉÖ´ÐÐÁËÓ°ÏóÌôÕ½ÊÂÎñ£¨ÈçÎïÌå³ö¾µÔÙÈë¾µ£©¡£¡£

2.Ó°Ïó¿É¿¿¶È (Memory Reliability, S_rel)£º½öÔÚÀֳɴ¥·¢ÌôÕ½µÄÑù±¾ÉÏÅÌËãÒ»ÖÂÐԵ÷Ö¡£¡£

×îÖÕµÄ M-Score È¡Á½Õߵĵ÷Çå¾²¾ùÊý£¬£¬£¬£¬Ö¼ÔÚ´¦·Öͨ¹ýÌìÉúÊØ¾É / ¾²Ì¬ÄÚÈÝÀ´¹æ±ÜÒ»ÖÂÐÔÌôÕ½µÄÐÐΪ£¬£¬£¬£¬½±ÀøÄÇЩ¼ÈÄÜÄ£Ä⶯̬ÌìÏÂÓÖÄܼá³ÖÒ»ÖÂÐÔµÄÄ£×Ó¡£¡£

¶Ô 14 ¸öÖ÷Á÷ SOTA Ä£×ÓµÄÆÀ²â·¢Ã÷

MBench ¶Ô 8 ¸öÎı¾Çý¶¯Ä£×ÓºÍ 6 ¸öÐж¯Çý¶¯Ä£×Ó¾ÙÐÐÁË´ó¹æÄ£ÆÀ²â£¬£¬£¬£¬ÆÀ²âЧ¹ûÅú×¢£¬£¬£¬£¬ÏÖÔÚ²¢²»±£´æ¼òµ¥Ä£×Ó¿ÉÒÔÔÚËùÓÐά¶ÈÉ϶¼ÌåÏÖ¾«²Ê£¬£¬£¬£¬Ó°ÏóÄÜÁ¦ÈÔÈ»ÊÇÁ÷ʽÊÓƵÌìÉúºÍÌìÏÂÄ£×ÓµÄÆÕ±éÆ¿¾±¡£¡£

Ä£×Ó·ÖÊýÀ×´ïͼ

ʵÑéÆÀ²âЧ¹û

¶ÔʵÑéЧ¹û¾ÙÐÐÆÊÎö£¬£¬£¬£¬»¹¿ÉÒÔ»ñµÃÈçÏ·¢Ã÷£º

¿Õ¼äÓëÒò¹ûÄÜÁ¦ÊÇÖ÷Ҫƿ¾±£ºÆÀ²âЧ¹ûÏÔʾ£¬£¬£¬£¬´ó¶¼Ä£×ÓÔÚ³¤Ê±ÐòÊÓ½Çת»»ÏµĿռ伸ºÎ»¹Ô­£¬£¬£¬£¬ÒÔ¼°Éæ¼°ÎïÀíÑÝ»¯µÄÒò¹ûÂß¼­¼á³ÖÉϱ£´æÏÔÖøȱÏÝ£¬£¬£¬£¬ÄÑÒÔÕÒ»ØÏûÊŵÄÊÓͼ»òÑÓÐø»­ÃæÍâµÄÎïÀíÀú³Ì¡£¡£Ðж¯Çý¶¯Ä£× ¡°Æ«¿Æ¡± Õ÷Ïó£ºÐж¯Ìõ¼þÄ£×ÓËäÈ»ÔÚ¿Õ¼äÎȹÌÐÔÖ¸±êÉÏÌåÏֽϺ㬣¬£¬£¬µ«±£´æÒ»¸öÆÕ±éµÄʧЧģʽ ¡ª¡ª ¼´ÇãÏòÓÚÌìÉúÌ«¹ý¾²Ì¬µÄ³¡¾°¡£¡£ÕâÖÖ·½Ê½ËäÈ»¹æ±ÜÁË¿Õ¼äÌ®Ëõ£¬£¬£¬£¬µ«Ä£×ÓÏÖʵÉÏÎÞ·¨Çý¶¯ÖØ´óµÄÎïÀíÑÝ»¯¡£¡£ÊÓ¾õ±ÆÕæ¶È²»¼´ÊÇÓ°ÏóÎȹÌÐÔ£ºÊµÑé֤ʵ£¬£¬£¬£¬Äܹ»ºÏ³É¸ßÖÊÁ¿¡¢¸ß±£Õæ»­ÃæµÄÄ£×Ó£¬£¬£¬£¬ÔÚ³¤Ê±ÐòÓ°Ïóά¶ÈÉÏ·×Æ綨ռÓÅ¡£¡£Õâ˵Ã÷´¿´âµÄÊÓ¾õÌìÉúÖ¸±êÎÞ·¨Ìæ»»¶ÔÄ£×ÓÄÚ²¿×´Ì¬³¤ÆÚÐÔµÄרÏîÆÀ¹À£¬£¬£¬£¬Ö¤ÊµÎú½¨ÉèÓ°ÏóÄÜÁ¦»ù×¼µÄÐëÒªÐÔ¡£¡£

´ÓÌìÉúµ¥ÕÅͼƬµ½ºÏÒòËØÖÓ¼¶ÊÓƵ£¬£¬£¬£¬ÊÓƵÌìÍâÐÐÒÕÒѾ­È¡µÃÁËÁîÈËÖõÄ¿µÄÇ°½ø¡£¡£È»¶ø£¬£¬£¬£¬ÒªÊµÏÖÄܹ»Ã÷È·¡¢Õ¹Íû²¢¾ÙÐн»»¥µÄÌìÏÂÄ£×Ó£¬£¬£¬£¬²¢ÈÃÁ÷ʽÊÓƵÌìÉú¼á³Ö³¤Ê±Ò»ÖÂÐÔ£¬£¬£¬£¬ÎÒÃÇÈÔÓкܳ¤µÄ·Ҫ×ß¡£¡£¶ø ¡°Ó°Ïó¡±£¬£¬£¬£¬ÕýÊǹ¹½¨ÕâЩ½¹µãÄÜÁ¦µÄ»ùʯ¡£¡£MBench Õ¹ÏÖÁËÄ¿½ñÖ÷Á÷Ä£×ÓÔÚÓ°ÏóÄÜÁ¦ÉϵÄÕæʵ½çÏߣ¬£¬£¬£¬Ò²ÎªÎ´À´µÄÑо¿Ö¸Ã÷ÎúÆ«Ïò¡£¡£

ÏÖÔÚ£¬£¬£¬£¬MBench ÒÑÖÜÈ«¿ªÔ´£¬£¬£¬£¬¿ªÔ´ÄÚÈÝ°üÀ¨£ºÍêÕûµÄ 1040 ¸öÆÀ²â°¸ÀýÊý¾Ý¼¯¡¢×Ô¶¯»¯ÆÀ²â´úÂëÓ빤¾ßÁ´¡¢ÊµÊ±¸üеĹûÕæÅÅÐаñ£¬£¬£¬£¬ÒÔ¼°ÏêϸµÄÊÖÒÕ±¨¸æÓëʵÑéЧ¹û¡£¡£ÏàÐÅÔÚ MBench µÄÍƶ¯Ï£¬£¬£¬£¬ÎÒÃÇÖÕ½«Ó­À´Äܹ» ¡°¼Ç×ÅÌìÏ¡¢Ã÷È·ÌìÏ¡¢Õ¹ÍûÌìÏ¡± µÄÏÂÒ»´úÊÓƵÌìÏÂÄ£×Ó¡£¡£

ÖйúÎÄÁª¸±Ö÷ϯ¸ßÊÀÃûÖ±ÑÔ£¬£¬£¬£¬ÕⲿӰϷдÀëÉ¢ÓëÐüÄ£¬£¬£¬¸üдͨË×È˵ļáÈ͸ßÉУ¬£¬£¬£¬ÊÇÏçÍÁÖйúµÄÉîÇé»ØÍû¡£¡£

¿­Ê±AG

ÃÀý³ÆÃÀÒÁ̸ÅÐÖØ»ØÕý¹ì