【imp-v1-3b】軽量MLLMがCAPTCHA認証に挑戦！果たして突破できるのか？

!pip install -U transformers
!pip install -q pillow accelerate einops

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image

torch.set_default_device("cuda")

# モデルとトークナイザー
model = AutoModelForCausalLM.from_pretrained(
    "MILVLG/imp-v1-3b", 
    torch_dtype=torch.float16, 
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "MILVLG/imp-v1-3b", 
    trust_remote_code=True
)

# プロンプトと画像
text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nDescribe the person in the image. ASSISTANT:"
image = Image.open("image.jpg")

# 推論
input_ids = tokenizer(text, return_tensors="pt").input_ids
image_tensor = model.image_preprocess(image)
output_ids = model.generate(
    input_ids,
    max_new_tokens=100,
    images=image_tensor,
    use_cache=True)[0]
print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())

「A chat between a curious user and an artificial intelligence assistant.
 The assistant gives helpful, detailed, and polite answers to the user's questions. 
USER: \nDescribe the person in the image. ASSISTANT:」

（好奇心旺盛なユーザーと人工知能アシスタントとのチャット。アシスタントはユーザーの質問に親切、詳しく、丁寧に答えてくれる。
ユーザー：「画像に写っている人物を教えてください。」アシスタント：）

The person in the image is a young man, possibly a boy, who is standing in front of a door. 
He is holding a brown box in his hands, and he appears to be bending over to get into the room.

日本語訳：
画像の人物は若い男性（おそらく少年）で、ドアの前に立っている。手には茶色の箱を持っており、部屋の中に入ろうと腰をかがめているように見える。

!pip install transformers

!pip install -q pillow accelerate einops

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer

from PIL import Image

torch.set_default_device("cuda")

#Create model

model = AutoModelForCausalLM.from_pretrained(

    "MILVLG/Imp-v1.5-3B-Phi2/", 

    torch_dtype=torch.float16, 

    device_map="auto",

    trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained("MILVLG/Imp-v1.5-3B-Phi2", trust_remote_code=True)

#Set inputs

text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are the colors of the bus in the image? ASSISTANT:"

image = Image.open("images/bus.jpg")

input_ids = tokenizer(text, return_tensors='pt').input_ids

image_tensor = model.image_preprocess(image)

#Generate the answer

output_ids = model.generate(

    input_ids,

    max_new_tokens=100,

    images=image_tensor,

    use_cache=True)[0]

print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())

!pip install transformers==4.37.2 accelerate torch torchvision pillow einops timm

import torch

from transformers import AutoModelForCausalLM, AutoTokenizer

from PIL import Image

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained(

    "MILVLG/imp-v1-3b", 

    torch_dtype=torch.float16, 

    device_map="auto",

    trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained("MILVLG/imp-v1-3b", trust_remote_code=True)

text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat text can you see in this image? Please tell me what is written. ASSISTANT:"

image = Image.open("/content/sample.png")

input_ids = tokenizer(text, return_tensors='pt').input_ids

image_tensor = model.image_preprocess(image)

output_ids = model.generate(

    input_ids,

    max_new_tokens=100,

    images=image_tensor,

    use_cache=True)[0]

print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())

The text in the image reads "The picture contains 8 characters."