{"id":1649,"date":"2024-07-12T08:51:59","date_gmt":"2024-07-12T08:51:59","guid":{"rendered":"https:\/\/webdot-alb.radisentech.com\/?post_type=publication&#038;p=1649"},"modified":"2024-08-29T02:13:20","modified_gmt":"2024-08-29T02:13:20","slug":"pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays","status":"publish","type":"publication","link":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/","title":{"rendered":"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays"},"content":{"rendered":"\n<div class=\"wp-block-columns is-layout-flex wp-container-core-columns-is-layout-1 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:33.33%\">\n<h2 class=\"wp-block-heading\" id=\"h-published\">Published<\/h2>\n\n\n\n<p>Medical Imaging with Deep Learning (2024)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-authors\">Authors<\/h2>\n\n\n\n<p>Yeongjae Cho<sup>1<\/sup>, Taehee Kim<sup>2<\/sup>, Heejun Shin<sup>2<\/sup>, Sungzoon Cho<sup>1<\/sup>, and Dongmyung Shin<sup>2<\/sup><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-affiliations\">Affiliations<\/h2>\n\n\n\n<p><em><sup>1<\/sup>Seoul National University<br><sup>2<\/sup>Artificial Intelligence Engineering Division, Radisen Co. Ltd<\/em><\/p>\n<\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\" style=\"flex-basis:66.66%\">\n<h2 class=\"wp-block-heading\" id=\"h-abstract\">Abstract<\/h2>\n\n\n\n<p>Difference visual question answering (diff-VQA) is a challenging task that requires answering complex questions based on differences between a pair of images. This task is particularly important in reading chest X-ray images because radiologists often compare multiple images of the same patient taken at different times to track disease progression and changes in its severity in their clinical practice. However, previous works focused on designing specific network architectures for the diff-VQA task, missing opportunities to enhance the model\u2019s performance using a pretrained vision-language model (VLM). Here, we introduce a novel VLM called PLURAL, which is pretrained on natural and longitudinal chest X-ray data for the diff-VQA task. The model is developed using a step-by-step approach, starting with being pretrained on natural images and texts, followed by being trained using longitudinal chest X-ray data. The longitudinal data consist of pairs of X-ray images, along with question-answer sets and radiologist\u2019s reports that describe the changes in lung abnormalities and diseases over time. Our experimental results show that the PLURAL model outperforms state-of-the-art methods not only in diff-VQA for longitudinal X-rays but also in conventional VQA for a single X-ray image. Through extensive experiments, we demonstrate the effectiveness of the proposed VLM architecture and pretraining method in improving the model\u2019s performance. Our code is available at: https:\/\/github.com\/yjch00\/PLURAL<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button is-style-fill\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/openreview.net\/forum?id=QTFVYcV92b\">Link to Publication<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<p><\/p>\n","protected":false},"featured_media":0,"template":"","meta":{"_acf_changed":false},"categories":[],"class_list":["post-1649","publication","type-publication","status-publish","hentry"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v22.3 (Yoast SEO v22.3) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays - Radisen<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/\" \/>\n<meta property=\"og:locale\" content=\"zh_CN\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays\" \/>\n<meta property=\"og:description\" content=\"Published Medical Imaging with Deep Learning (2024) Aut [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/\" \/>\n<meta property=\"og:site_name\" content=\"Radisen\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-29T02:13:20+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.radisentech.com\/wp-content\/uploads\/2024\/06\/Site-image.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"675\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 \u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/\",\"url\":\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/\",\"name\":\"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays - Radisen\",\"isPartOf\":{\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/#website\"},\"datePublished\":\"2024-07-12T08:51:59+00:00\",\"dateModified\":\"2024-08-29T02:13:20+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/#breadcrumb\"},\"inLanguage\":\"zh-Hans\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.radisentech.com\/zh-hans\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/#website\",\"url\":\"https:\/\/www.radisentech.com\/zh-hans\/\",\"name\":\"Radisen\",\"description\":\"AI\ub97c \ud1b5\ud55c \uc758\ub8cc \ud601\uc2e0\uc73c\ub85c \uac74\uac15\ud55c \uc0b6\uc758 \uac00\uce58\ub97c \ub192\uc785\ub2c8\ub2e4.\",\"publisher\":{\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.radisentech.com\/zh-hans\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"zh-Hans\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/#organization\",\"name\":\"Radisen\",\"url\":\"https:\/\/www.radisentech.com\/zh-hans\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"zh-Hans\",\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.radisentech.com\/wp-content\/uploads\/2024\/06\/favicon.png\",\"contentUrl\":\"https:\/\/www.radisentech.com\/wp-content\/uploads\/2024\/06\/favicon.png\",\"width\":512,\"height\":512,\"caption\":\"Radisen\"},\"image\":{\"@id\":\"https:\/\/www.radisentech.com\/zh-hans\/#\/schema\/logo\/image\/\"}}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays - Radisen","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/","og_locale":"zh_CN","og_type":"article","og_title":"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays","og_description":"Published Medical Imaging with Deep Learning (2024) Aut [&hellip;]","og_url":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/","og_site_name":"Radisen","article_modified_time":"2024-08-29T02:13:20+00:00","og_image":[{"width":1200,"height":675,"url":"https:\/\/www.radisentech.com\/wp-content\/uploads\/2024\/06\/Site-image.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_misc":{"\u9884\u8ba1\u9605\u8bfb\u65f6\u95f4":"2 \u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/","url":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/","name":"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays - Radisen","isPartOf":{"@id":"https:\/\/www.radisentech.com\/zh-hans\/#website"},"datePublished":"2024-07-12T08:51:59+00:00","dateModified":"2024-08-29T02:13:20+00:00","breadcrumb":{"@id":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/#breadcrumb"},"inLanguage":"zh-Hans","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.radisentech.com\/zh-hans\/publication\/pretraining-vision-language-model-for-difference-visual-question-answering-in-longitudinal-chest-x-rays\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.radisentech.com\/zh-hans\/"},{"@type":"ListItem","position":2,"name":"Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays"}]},{"@type":"WebSite","@id":"https:\/\/www.radisentech.com\/zh-hans\/#website","url":"https:\/\/www.radisentech.com\/zh-hans\/","name":"Radisen","description":"AI\ub97c \ud1b5\ud55c \uc758\ub8cc \ud601\uc2e0\uc73c\ub85c \uac74\uac15\ud55c \uc0b6\uc758 \uac00\uce58\ub97c \ub192\uc785\ub2c8\ub2e4.","publisher":{"@id":"https:\/\/www.radisentech.com\/zh-hans\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.radisentech.com\/zh-hans\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"zh-Hans"},{"@type":"Organization","@id":"https:\/\/www.radisentech.com\/zh-hans\/#organization","name":"Radisen","url":"https:\/\/www.radisentech.com\/zh-hans\/","logo":{"@type":"ImageObject","inLanguage":"zh-Hans","@id":"https:\/\/www.radisentech.com\/zh-hans\/#\/schema\/logo\/image\/","url":"https:\/\/www.radisentech.com\/wp-content\/uploads\/2024\/06\/favicon.png","contentUrl":"https:\/\/www.radisentech.com\/wp-content\/uploads\/2024\/06\/favicon.png","width":512,"height":512,"caption":"Radisen"},"image":{"@id":"https:\/\/www.radisentech.com\/zh-hans\/#\/schema\/logo\/image\/"}}]}},"_links":{"self":[{"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/publication\/1649","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/publication"}],"about":[{"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/types\/publication"}],"version-history":[{"count":5,"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/publication\/1649\/revisions"}],"predecessor-version":[{"id":3338,"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/publication\/1649\/revisions\/3338"}],"wp:attachment":[{"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/media?parent=1649"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.radisentech.com\/zh-hans\/wp-json\/wp\/v2\/categories?post=1649"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}