딥러닝 공부방

폴리곤을 segmentation mask로 변환하기(Polygon to mask)

AI 꿈나무 — Fri, 23 Dec 2022 19:19:11 +0900

폴리곤 좌표로 표현되어 있는 mask를 binary mask로 변환하는 방법을 알아보겠다.

구글링 해보니 잘 안나와서 한번 작성해본다.

여러 자료를 찾아봤는데 skimage.draw.polygon2mask 가 제일 편한것 같다.

from skimage.draw import polygon2mask 로 함수를 불러와서

image_shape와 np.array 타입의 polygon을 넣어주면 된다.

https://scikit-image.org/docs/stable/api/skimage.draw.html#skimage.draw.polygon2mask

Module: draw — skimage v0.19.2 docs

The range of values to sample pixel values from. For grayscale images the format is (min, max). For multichannel - ((min, max),) if the ranges are equal across the channels, and ((min_0, max_0), … (min_N, max_N)) if they differ. As the function supports

scikit-image.org

segmentation mask 덩어리 갯수 확인하기

AI 꿈나무 — Sun, 21 Aug 2022 20:40:29 +0900

위 그림은 mask가 두 덩어리로 이루어 있습니다.

이 케이스를 파악하는 코드를 짜보았는뎅 공유합니다.

def count_contours(masks, threshold=1000):
    if type(masks[0]) == torch.Tensor:
        masks = [mask.cpu().numpy() for mask in masks]

    counts = []

    for mask in masks:
        count = 0
        contours, _ = cv2.findContours(mask.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
        for contour in contours:
            area = cv2.contourArea(contour)
            if area >= threshold:
                count += 1


        counts.append(count)

    return np.mean(counts)

cv2.findContours 함수를 사용하여 contour를 얻어옵니다.

contourArea 함수를 이용하여 contour의 영역 크기를 계산한 후, threshold 이상인 경우만 카운트 합니다.

Segmentation mask의 center point 계산하기

AI 꿈나무 — Sun, 21 Aug 2022 20:36:00 +0900

Segmentation mask의 center point를 얻어오는게 필요해서 코드를 짜 보았습니다.

mask는 0 또는 1의 값을 갖고 있으므로 numpy 혹은 pytorch의 nonzero 함수를 사용사용하면 됩니다.

coordinates = np.nonzero(mask)

x_coordinates = coordinates[0]
y_coordinates = coordinates[1]

x_min, x_max, y_min, y_max = np.min(x_coordinates), np.max(x_coordinates), np.min(y_coordinates), np.max(y_coordinates)

x_center, y_center = (x_max + x_min) / 2, (y_max + y_min) / 2

어떻게 center point를 계산할지 고민하다가 위 처럼 짜보았는데, 혹시 도움이 필요하신 분이 계실까봐 공유합니다

Self-supervised Learning에 대하여

AI 꿈나무 — Wed, 17 Aug 2022 12:08:40 +0900

심심해서 적어보는 글.

Self-supervised learning은 데이터가 부족한 환경에서 사용하는 것이 아니라, 데이터는 많은데 annotation이 없는 경우에 사용하는 것이다.

unlabeled data 로부터 어떻게 pretask를 만들어서 효율적인 representation을 뽑아내느냐가 중요하며, 작년 까지 핫했던 SSL 모델들(DINO, MoCO, SimCL?)은 unlabeled data에 aumentation에 강하게 줘서 contrastive learning으로 augmentation에 불변한 representation을 뽑아내는 방향으로 발전해왔다. 22년 SSL 논문은 안읽어봐서 모르겠는데 현재도 비슷한 방향으로 연구가 진행되고 있지 않을까 싶다.

또한 SSL방법론이 성능을 내기 위해서는 supervision보다 더 많은 데이터를 필요로 할 수 있다. 중요한 것은 데이터가 부족한 상황에서 SSL을 사용하는게 아니라, annotation이 없는 상황에서 사용하는 것. unlabeled data로부터 SSL로 학습한 모델을 약간의 annotation data로 fine-tuning해서 높은 성능을 달성하자!

[PyTorch] CLIP의 text encoder에는 attention mask가 존재합니다.

AI 꿈나무 — Mon, 1 Aug 2022 15:30:57 +0900

CLIP의 text encoder 내부의 Multi_head_attention에서 attention mask가 None으로 입력되는 줄 알았다.

아이디어를 구현하기 위해 attention mask 부분을 만들어서 넣어줬더니.. 성능이 엄청 떨어졌다.

한번 확인해보니 CLIP의 text encoder에서 attention mask가 None으로 입력되는 것이 아니라

다음과 같이 들어간다. 근데 어떤 layer에서는 None으로 들어가기 때문에 구체적으로 확인해볼 필요는 있다. 항상 저렇게 attention mask가 들어가는게 아니다.

BERT 구조임에도 autoregressive하게 문장을 보게 하려는 의도 인듯?

[PyTorch] Multi_head_attention에서 target sequence length와 source sequence length 의미

AI 꿈나무 — Wed, 27 Jul 2022 20:23:31 +0900

Multi_head_attention에서 target sequence length와 source sequence length 의미

연구를 위해 pytorch의 multi head attention에 attention mask를 씌워줘야 했다.

도큐먼트를 보면 L은 target sequence length를 의미하고 S는 source sequence length를 말하는데, 이 둘은 무엇일까?

pytorch 내부 코드를 뜯어보니 target sequence length는 query의 길이를 의미한다.

soure sequence length는 key의 길이를 의미함.

구글링해도 관련 내용을 찾기 어려워서 작성해본당. 나만 모르고 다 아는 내용이라서 구글링해도 못찾았던 거일수도?

[PyTorch] Tensor.retain_grad()

AI 꿈나무 — Tue, 19 Jul 2022 19:39:00 +0900

a.reatain_grad()를 통해 gradient가 사라지는 것을 예방할 수 있다.

계산그래프에서 leaf node가 아닌 tensor의 gradient는 계산 후 날라가는데, retain_grad를 통해 날라가지 않고 붙잡을 수 있다.

https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/

Debugging and Visualisation in PyTorch using Hooks

In this post, we cover debugging and Visualisation in PyTorch. We go over PyTorch hooks and how to use them to debug our backpass, visualise activations and modify gradients.

blog.paperspace.com

[PyTorch] register_hook을 사용하여 Transformer 내부의 Attention matrix(Torch.Tensor)의 gradient 받아오기

AI 꿈나무 — Thu, 14 Jul 2022 17:27:27 +0900

register_hook을 사용하여 Transformer 내부의 Attention matrix(Torch.Tensor)의 gradient 받아오기

모델의 파라미터에 대한 grad가 아닌, Tensor object에 대한 grad는 계산만하고 날라가버린다. 즉, loss.backward()를 통해 backpropagation을 진행하면 중간 연산에 필요한 Tensor 변수의 gradient는 .grad로 저장이 안되고 계산이 끝나면 날라간다는 말이다. 따라서 Tensor object에 register_hook 함수로 gradient를 한번 붙잡아야 한다. 붙잡는다는 말은 gradient가 계산되었을 때, 날라가도록 두는게 아니라 다른 변수에 저장해야 한다는 말이다.

나는 중간 연산의 Tensor object에 가해지는 gradient값이 필요했다.

Transformer의 Query와 Key의 행렬곱을 통해 attention matrix를 얻는데, backpropagation과정에서 attention matrix에 대한 gradient를 얻는 방법을 공유하려고 한다.

Transformer는 Torch 내부 class인 MultiheadAttention을 사용하여 구현되는데, 이 내부 코드를 수정해줘야 한다.

Torch 내부 코드를 수정하기가 껄끄러우면, MultiheadAttention class와 이 class가 실행되기 위해 필요한 함수들을 따로 utils.py로 옮겨서 utils.MultiheadAttention class로 Transformer 구조를 변경해주면 된다. 즉, Torch의 class를 사용하는게 아니라 Torch의 class와 동일하지만, util에서 class를 꺼내서 사용하는 것이다.

Torch의 MultiheadAttention class를 살펴보면 F.multi_head_attention_forward 함수가 실행되고

F.multi_head_attention_forward 내부에는 _scaled_dot_product_attention 함수를 통해 attention matrix가 계산된다.

attention matrix가 계산되는 함수 내부까지 내려가서 attention matrix에 register_hook을 걸어줘야 한다.

# _scaled_dot_product_attention 함수 내부
B, Nt, E = q.shape
q = q / math.sqrt(E)
# (B, Nt, E) x (B, E, Ns) -> (B, Nt, Ns)
attn = torch.bmm(q, k.transpose(-2, -1))
self.save_attn_map(attn) # modified
if attn_mask is not None:
    attn += attn_mask
attn = softmax(attn, dim=-1)
if dropout_p > 0.0:
    attn = dropout(attn, p=dropout_p)
# (B, Nt, Ns) x (B, Ns, E) -> (B, Nt, E)
output = torch.bmm(attn, v)

gradient_hook = attn.register_hook(self.save_attn_gradient) 

def save_attn_gradient(self, gradient): # modified
    gradient = gradient.view(self.bsz, self.num_heads, self.tgt_len, self.src_len)
    gradient = gradient.sum(dim=1) / self.num_heads
    self.attention_map_gradients = gradient

Torch.Tensor 변수에 register_hook을 걸어두면 gradient 값만을 받아올 수 있다. gradient값을 받아와서 self.attention_map_gradients에 gradient 값을 저장하는 함수를 register_hook에 넣어줬당

self 인자를 사용하려면 class 내부에 _scaled_dot_product_attention 함수가 존재해야 하는데, Torch 내부 함수는 class 내부에 함수가 구현되지 않아서 나는 util.py 폴더에 Multi_head_attention class 내부에 _scaled_dot_product_attention 함수를 넣어줬다.

나는 attention map도 받아오는게 필요해서 다음의 함수도 class 내부에 넣어줬고, attention map과 attention map의 gradient를 받아오는 함수도 class 내부에 넣어줬당

    def save_attn_map(self, attn):
        attn = attn.view(self.bsz, self.num_heads, self.tgt_len, self.src_len)
        attn = attn.sum(dim=1) / self.num_heads
        self.attention_map = attn

    def get_attn_map(self):
        return self.attention_map

    def get_attn_gradients(self):
        return self.attention_map_gradients

그러면 어떻게 attention map와 attention map의 gradient를 받아오냐면, Transformer 모델의 Multi_head_attention class에 접근해서 get_attn_map과 get_attn_gradients 함수를 실행해주면 된다.

attention_map 정보는 forward를 통해 저장되고, attention_map의 gradient는 loss.backward를 실행하면 Multi_head_attention class 내부의 self.attention_map_gradient 변수에 gradient가 저장된다.

for blk in clip_model.model.transformer.resblocks:
    cam = blk.attn.get_attn_map()
    gradients = blk.attn.get_attn_gradients()
    # gradients = blk.get_attn_gradients()
    print(cam.shape)
    print(gradients.shape)
    break

전체 코드

class MHA(Module):
    r"""Allows the model to jointly attend to information
    from different representation subspaces.
    See `Attention Is All You Need <https://arxiv.org/abs/1706.03762>`_.

    .. math::
        \text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O

    where :math:`head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)`.

    Args:
        embed_dim: Total dimension of the model.
        num_heads: Number of parallel attention heads. Note that ``embed_dim`` will be split
            across ``num_heads`` (i.e. each head will have dimension ``embed_dim // num_heads``).
        dropout: Dropout probability on ``attn_output_weights``. Default: ``0.0`` (no dropout).
        bias: If specified, adds bias to input / output projection layers. Default: ``True``.
        add_bias_kv: If specified, adds bias to the key and value sequences at dim=0. Default: ``False``.
        add_zero_attn: If specified, adds a new batch of zeros to the key and value sequences at dim=1.
            Default: ``False``.
        kdim: Total number of features for keys. Default: ``None`` (uses ``kdim=embed_dim``).
        vdim: Total number of features for values. Default: ``None`` (uses ``vdim=embed_dim``).
        batch_first: If ``True``, then the input and output tensors are provided
            as (batch, seq, feature). Default: ``False`` (seq, batch, feature).

    Examples::

        >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads)
        >>> attn_output, attn_output_weights = multihead_attn(query, key, value)
    """
    __constants__ = ['batch_first']
    bias_k: Optional[torch.Tensor]
    bias_v: Optional[torch.Tensor]

    def __init__(self, embed_dim, num_heads, dropout=0., bias=True, add_bias_kv=False, add_zero_attn=False,
                 kdim=None, vdim=None, batch_first=False, device=None, dtype=None) -> None:
        factory_kwargs = {'device': device, 'dtype': dtype}
        super(MHA, self).__init__()
        self.embed_dim = embed_dim
        self.kdim = kdim if kdim is not None else embed_dim
        self.vdim = vdim if vdim is not None else embed_dim
        self._qkv_same_embed_dim = self.kdim == embed_dim and self.vdim == embed_dim

        self.num_heads = num_heads
        self.dropout = dropout
        self.batch_first = batch_first
        self.head_dim = embed_dim // num_heads
        assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"

        if self._qkv_same_embed_dim is False:
            self.q_proj_weight = Parameter(torch.empty((embed_dim, embed_dim), **factory_kwargs))
            self.k_proj_weight = Parameter(torch.empty((embed_dim, self.kdim), **factory_kwargs))
            self.v_proj_weight = Parameter(torch.empty((embed_dim, self.vdim), **factory_kwargs))
            self.register_parameter('in_proj_weight', None)
        else:
            self.in_proj_weight = Parameter(torch.empty((3 * embed_dim, embed_dim), **factory_kwargs))
            self.register_parameter('q_proj_weight', None)
            self.register_parameter('k_proj_weight', None)
            self.register_parameter('v_proj_weight', None)

        if bias:
            self.in_proj_bias = Parameter(torch.empty(3 * embed_dim, **factory_kwargs))
        else:
            self.register_parameter('in_proj_bias', None)
        self.out_proj = NonDynamicallyQuantizableLinear(embed_dim, embed_dim, bias=bias, **factory_kwargs)

        if add_bias_kv:
            self.bias_k = Parameter(torch.empty((1, 1, embed_dim), **factory_kwargs))
            self.bias_v = Parameter(torch.empty((1, 1, embed_dim), **factory_kwargs))
        else:
            self.bias_k = self.bias_v = None

        self.add_zero_attn = add_zero_attn

        self._reset_parameters()

        self.attention_map = None
        self.attention_map_gradients = None

    def _reset_parameters(self):
        if self._qkv_same_embed_dim:
            xavier_uniform_(self.in_proj_weight)
        else:
            xavier_uniform_(self.q_proj_weight)
            xavier_uniform_(self.k_proj_weight)
            xavier_uniform_(self.v_proj_weight)

        if self.in_proj_bias is not None:
            constant_(self.in_proj_bias, 0.)
            constant_(self.out_proj.bias, 0.)
        if self.bias_k is not None:
            xavier_normal_(self.bias_k)
        if self.bias_v is not None:
            xavier_normal_(self.bias_v)

    def __setstate__(self, state):
        # Support loading old MultiheadAttention checkpoints generated by v1.1.0
        if '_qkv_same_embed_dim' not in state:
            state['_qkv_same_embed_dim'] = True

        super(MHA, self).__setstate__(state)

    def forward(self, query: Tensor, key: Tensor, value: Tensor, key_padding_mask: Optional[Tensor] = None,
                need_weights: bool = True, attn_mask: Optional[Tensor] = None) -> Tuple[Tensor, Optional[Tensor]]:
        r"""
    Args:
        query: Query embeddings of shape :math:`(L, N, E_q)` when ``batch_first=False`` or :math:`(N, L, E_q)`
            when ``batch_first=True``, where :math:`L` is the target sequence length, :math:`N` is the batch size,
            and :math:`E_q` is the query embedding dimension ``embed_dim``. Queries are compared against
            key-value pairs to produce the output. See "Attention Is All You Need" for more details.
        key: Key embeddings of shape :math:`(S, N, E_k)` when ``batch_first=False`` or :math:`(N, S, E_k)` when
            ``batch_first=True``, where :math:`S` is the source sequence length, :math:`N` is the batch size, and
            :math:`E_k` is the key embedding dimension ``kdim``. See "Attention Is All You Need" for more details.
        value: Value embeddings of shape :math:`(S, N, E_v)` when ``batch_first=False`` or :math:`(N, S, E_v)` when
            ``batch_first=True``, where :math:`S` is the source sequence length, :math:`N` is the batch size, and
            :math:`E_v` is the value embedding dimension ``vdim``. See "Attention Is All You Need" for more details.
        key_padding_mask: If specified, a mask of shape :math:`(N, S)` indicating which elements within ``key``
            to ignore for the purpose of attention (i.e. treat as "padding"). Binary and byte masks are supported.
            For a binary mask, a ``True`` value indicates that the corresponding ``key`` value will be ignored for
            the purpose of attention. For a byte mask, a non-zero value indicates that the corresponding ``key``
            value will be ignored.
        need_weights: If specified, returns ``attn_output_weights`` in addition to ``attn_outputs``.
            Default: ``True``.
        attn_mask: If specified, a 2D or 3D mask preventing attention to certain positions. Must be of shape
            :math:`(L, S)` or :math:`(N\cdot\text{num\_heads}, L, S)`, where :math:`N` is the batch size,
            :math:`L` is the target sequence length, and :math:`S` is the source sequence length. A 2D mask will be
            broadcasted across the batch while a 3D mask allows for a different mask for each entry in the batch.
            Binary, byte, and float masks are supported. For a binary mask, a ``True`` value indicates that the
            corresponding position is not allowed to attend. For a byte mask, a non-zero value indicates that the
            corresponding position is not allowed to attend. For a float mask, the mask values will be added to
            the attention weight.

    Outputs:
        - **attn_output** - Attention outputs of shape :math:`(L, N, E)` when ``batch_first=False`` or
          :math:`(N, L, E)` when ``batch_first=True``, where :math:`L` is the target sequence length, :math:`N` is
          the batch size, and :math:`E` is the embedding dimension ``embed_dim``.
        - **attn_output_weights** - Attention output weights of shape :math:`(N, L, S)`, where :math:`N` is the batch
          size, :math:`L` is the target sequence length, and :math:`S` is the source sequence length. Only returned
          when ``need_weights=True``.
        """
        if self.batch_first:
            query, key, value = [x.transpose(1, 0) for x in (query, key, value)]

        if not self._qkv_same_embed_dim:
            attn_output, attn_output_weights = self.multi_head_attention_forward(
                query, key, value, self.embed_dim, self.num_heads,
                self.in_proj_weight, self.in_proj_bias,
                self.bias_k, self.bias_v, self.add_zero_attn,
                self.dropout, self.out_proj.weight, self.out_proj.bias,
                training=self.training,
                key_padding_mask=key_padding_mask, need_weights=need_weights,
                attn_mask=attn_mask, use_separate_proj_weight=True,
                q_proj_weight=self.q_proj_weight, k_proj_weight=self.k_proj_weight,
                v_proj_weight=self.v_proj_weight)
        else:
            attn_output, attn_output_weights = self.multi_head_attention_forward(
                query, key, value, self.embed_dim, self.num_heads,
                self.in_proj_weight, self.in_proj_bias,
                self.bias_k, self.bias_v, self.add_zero_attn,
                self.dropout, self.out_proj.weight, self.out_proj.bias,
                training=self.training,
                key_padding_mask=key_padding_mask, need_weights=need_weights,
                attn_mask=attn_mask)
        if self.batch_first:
            return attn_output.transpose(1, 0), attn_output_weights
        else:
            return attn_output, attn_output_weights


    def multi_head_attention_forward(self,
        query: Tensor,
        key: Tensor,
        value: Tensor,
        embed_dim_to_check: int,
        num_heads: int,
        in_proj_weight: Tensor,
        in_proj_bias: Optional[Tensor],
        bias_k: Optional[Tensor],
        bias_v: Optional[Tensor],
        add_zero_attn: bool,
        dropout_p: float,
        out_proj_weight: Tensor,
        out_proj_bias: Optional[Tensor],
        training: bool = True,
        key_padding_mask: Optional[Tensor] = None,
        need_weights: bool = True,
        attn_mask: Optional[Tensor] = None,
        use_separate_proj_weight: bool = False,
        q_proj_weight: Optional[Tensor] = None,
        k_proj_weight: Optional[Tensor] = None,
        v_proj_weight: Optional[Tensor] = None,
        static_k: Optional[Tensor] = None,
        static_v: Optional[Tensor] = None,
    ) -> Tuple[Tensor, Optional[Tensor]]:

        r"""
        Args:
            query, key, value: map a query and a set of key-value pairs to an output.
                See "Attention Is All You Need" for more details.
            embed_dim_to_check: total dimension of the model.
            num_heads: parallel attention heads.
            in_proj_weight, in_proj_bias: input projection weight and bias.
            bias_k, bias_v: bias of the key and value sequences to be added at dim=0.
            add_zero_attn: add a new batch of zeros to the key and
                           value sequences at dim=1.
            dropout_p: probability of an element to be zeroed.
            out_proj_weight, out_proj_bias: the output projection weight and bias.
            training: apply dropout if is ``True``.
            key_padding_mask: if provided, specified padding elements in the key will
                be ignored by the attention. This is an binary mask. When the value is True,
                the corresponding value on the attention layer will be filled with -inf.
            need_weights: output attn_output_weights.
            attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all
                the batches while a 3D mask allows to specify a different mask for the entries of each batch.
            use_separate_proj_weight: the function accept the proj. weights for query, key,
                and value in different forms. If false, in_proj_weight will be used, which is
                a combination of q_proj_weight, k_proj_weight, v_proj_weight.
            q_proj_weight, k_proj_weight, v_proj_weight, in_proj_bias: input projection weight and bias.
            static_k, static_v: static key and value used for attention operators.


        Shape:
            Inputs:
            - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is
              the embedding dimension.
            - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is
              the embedding dimension.
            - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is
              the embedding dimension.
            - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length.
              If a ByteTensor is provided, the non-zero positions will be ignored while the zero positions
              will be unchanged. If a BoolTensor is provided, the positions with the
              value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged.
            - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length.
              3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length,
              S is the source sequence length. attn_mask ensures that position i is allowed to attend the unmasked
              positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend
              while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True``
              are not allowed to attend while ``False`` values will be unchanged. If a FloatTensor
              is provided, it will be added to the attention weight.
            - static_k: :math:`(N*num_heads, S, E/num_heads)`, where S is the source sequence length,
              N is the batch size, E is the embedding dimension. E/num_heads is the head dimension.
            - static_v: :math:`(N*num_heads, S, E/num_heads)`, where S is the source sequence length,
              N is the batch size, E is the embedding dimension. E/num_heads is the head dimension.

            Outputs:
            - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size,
              E is the embedding dimension.
            - attn_output_weights: :math:`(N, L, S)` where N is the batch size,
              L is the target sequence length, S is the source sequence length.
        """
        # set up shape vars
        tgt_len, bsz, embed_dim = query.shape
        self.tgt_len, self.bsz, self.embed_dim = tgt_len, bsz, embed_dim

        src_len, _, _ = key.shape
        self.src_len = src_len

        assert embed_dim == embed_dim_to_check, \
            f"was expecting embedding dimension of {embed_dim_to_check}, but got {embed_dim}"
        if isinstance(embed_dim, torch.Tensor):
            # embed_dim can be a tensor when JIT tracing
            head_dim = embed_dim.div(num_heads, rounding_mode='trunc')
        else:
            head_dim = embed_dim // num_heads
        assert head_dim * num_heads == embed_dim, f"embed_dim {embed_dim} not divisible by num_heads {num_heads}"
        if use_separate_proj_weight:
            # allow MHA to have different embedding dimensions when separate projection weights are used
            assert key.shape[:2] == value.shape[:2], \
                f"key's sequence and batch dims {key.shape[:2]} do not match value's {value.shape[:2]}"
        else:
            assert key.shape == value.shape, f"key shape {key.shape} does not match value shape {value.shape}"

        #
        # compute in-projection
        #
        if not use_separate_proj_weight:
            q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias)
        else:
            assert q_proj_weight is not None, "use_separate_proj_weight is True but q_proj_weight is None"
            assert k_proj_weight is not None, "use_separate_proj_weight is True but k_proj_weight is None"
            assert v_proj_weight is not None, "use_separate_proj_weight is True but v_proj_weight is None"
            if in_proj_bias is None:
                b_q = b_k = b_v = None
            else:
                b_q, b_k, b_v = in_proj_bias.chunk(3)
            q, k, v = _in_projection(query, key, value, q_proj_weight, k_proj_weight, v_proj_weight, b_q, b_k, b_v)

        # prep attention mask
        if attn_mask is not None:
            if attn_mask.dtype == torch.uint8:
                warnings.warn("Byte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
                attn_mask = attn_mask.to(torch.bool)
            else:
                assert attn_mask.is_floating_point() or attn_mask.dtype == torch.bool, \
                    f"Only float, byte, and bool types are supported for attn_mask, not {attn_mask.dtype}"
            # ensure attn_mask's dim is 3
            if attn_mask.dim() == 2:
                correct_2d_size = (tgt_len, src_len)
                if attn_mask.shape != correct_2d_size:
                    raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
                attn_mask = attn_mask.unsqueeze(0)
            elif attn_mask.dim() == 3:
                correct_3d_size = (bsz * num_heads, tgt_len, src_len)
                if attn_mask.shape != correct_3d_size:
                    raise RuntimeError(f"The shape of the 3D attn_mask is {attn_mask.shape}, but should be {correct_3d_size}.")
            else:
                raise RuntimeError(f"attn_mask's dimension {attn_mask.dim()} is not supported")

        # prep key padding mask
        if key_padding_mask is not None and key_padding_mask.dtype == torch.uint8:
            warnings.warn("Byte tensor for key_padding_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.")
            key_padding_mask = key_padding_mask.to(torch.bool)

        # add bias along batch dimension (currently second)
        if bias_k is not None and bias_v is not None:
            assert static_k is None, "bias cannot be added to static key."
            assert static_v is None, "bias cannot be added to static value."
            k = torch.cat([k, bias_k.repeat(1, bsz, 1)])
            v = torch.cat([v, bias_v.repeat(1, bsz, 1)])
            if attn_mask is not None:
                attn_mask = pad(attn_mask, (0, 1))
            if key_padding_mask is not None:
                key_padding_mask = pad(key_padding_mask, (0, 1))
        else:
            assert bias_k is None
            assert bias_v is None

        #
        # reshape q, k, v for multihead attention and make em batch first
        #
        q = q.contiguous().view(tgt_len, bsz * num_heads, head_dim).transpose(0, 1)
        if static_k is None:
            k = k.contiguous().view(k.shape[0], bsz * num_heads, head_dim).transpose(0, 1)
        else:
            # TODO finish disentangling control flow so we don't do in-projections when statics are passed
            assert static_k.size(0) == bsz * num_heads, \
                f"expecting static_k.size(0) of {bsz * num_heads}, but got {static_k.size(0)}"
            assert static_k.size(2) == head_dim, \
                f"expecting static_k.size(2) of {head_dim}, but got {static_k.size(2)}"
            k = static_k
        if static_v is None:
            v = v.contiguous().view(v.shape[0], bsz * num_heads, head_dim).transpose(0, 1)
        else:
            # TODO finish disentangling control flow so we don't do in-projections when statics are passed
            assert static_v.size(0) == bsz * num_heads, \
                f"expecting static_v.size(0) of {bsz * num_heads}, but got {static_v.size(0)}"
            assert static_v.size(2) == head_dim, \
                f"expecting static_v.size(2) of {head_dim}, but got {static_v.size(2)}"
            v = static_v

        # add zero attention along batch dimension (now first)
        if add_zero_attn:
            zero_attn_shape = (bsz * num_heads, 1, head_dim)
            k = torch.cat([k, torch.zeros(zero_attn_shape, dtype=k.dtype, device=k.device)], dim=1)
            v = torch.cat([v, torch.zeros(zero_attn_shape, dtype=v.dtype, device=v.device)], dim=1)
            if attn_mask is not None:
                attn_mask = pad(attn_mask, (0, 1))
            if key_padding_mask is not None:
                key_padding_mask = pad(key_padding_mask, (0, 1))

        # update source sequence length after adjustments
        src_len = k.size(1)

        # merge key padding and attention masks
        if key_padding_mask is not None:
            assert key_padding_mask.shape == (bsz, src_len), \
                f"expecting key_padding_mask shape of {(bsz, src_len)}, but got {key_padding_mask.shape}"
            key_padding_mask = key_padding_mask.view(bsz, 1, 1, src_len).   \
                expand(-1, num_heads, -1, -1).reshape(bsz * num_heads, 1, src_len)
            if attn_mask is None:
                attn_mask = key_padding_mask
            elif attn_mask.dtype == torch.bool:
                attn_mask = attn_mask.logical_or(key_padding_mask)
            else:
                attn_mask = attn_mask.masked_fill(key_padding_mask, float("-inf"))

        # convert mask to float
        if attn_mask is not None and attn_mask.dtype == torch.bool:
            new_attn_mask = torch.zeros_like(attn_mask, dtype=torch.float)
            new_attn_mask.masked_fill_(attn_mask, float("-inf"))
            attn_mask = new_attn_mask

        # adjust dropout probability
        if not training:
            dropout_p = 0.0

        #
        # (deep breath) calculate attention and out projection
        #
        attn_output, attn_output_weights = self._scaled_dot_product_attention(q, k, v, attn_mask, dropout_p)
        attn_output = attn_output.transpose(0, 1).contiguous().view(tgt_len, bsz, embed_dim)
        attn_output = linear(attn_output, out_proj_weight, out_proj_bias)

        if need_weights:
            # average attention weights over heads
            attn_output_weights = attn_output_weights.view(bsz, num_heads, tgt_len, src_len)

            return attn_output, attn_output_weights.sum(dim=1) / num_heads
        else:
            return attn_output, None

    def _scaled_dot_product_attention(self,
            q: Tensor,
            k: Tensor,
            v: Tensor,
            attn_mask: Optional[Tensor] = None,
            dropout_p: float = 0.0,
    ) -> Tuple[Tensor, Tensor]:
        r"""
        Computes scaled dot product attention on query, key and value tensors, using
        an optional attention mask if passed, and applying dropout if a probability
        greater than 0.0 is specified.
        Returns a tensor pair containing attended values and attention weights.

        Args:
            q, k, v: query, key and value tensors. See Shape section for shape details.
            attn_mask: optional tensor containing mask values to be added to calculated
                attention. May be 2D or 3D; see Shape section for details.
            dropout_p: dropout probability. If greater than 0.0, dropout is applied.

        Shape:
            - q: :math:`(B, Nt, E)` where B is batch size, Nt is the target sequence length,
                and E is embedding dimension.
            - key: :math:`(B, Ns, E)` where B is batch size, Ns is the source sequence length,
                and E is embedding dimension.
            - value: :math:`(B, Ns, E)` where B is batch size, Ns is the source sequence length,
                and E is embedding dimension.
            - attn_mask: either a 3D tensor of shape :math:`(B, Nt, Ns)` or a 2D tensor of
                shape :math:`(Nt, Ns)`.

            - Output: attention values have shape :math:`(B, Nt, E)`; attention weights
                have shape :math:`(B, Nt, Ns)`
        """


        B, Nt, E = q.shape
        q = q / math.sqrt(E)
        # (B, Nt, E) x (B, E, Ns) -> (B, Nt, Ns)
        attn = torch.bmm(q, k.transpose(-2, -1))
        self.save_attn_map(attn) # modified
        if attn_mask is not None:
            attn += attn_mask
        attn = softmax(attn, dim=-1)
        if dropout_p > 0.0:
            attn = dropout(attn, p=dropout_p)
        # (B, Nt, Ns) x (B, Ns, E) -> (B, Nt, E)
        output = torch.bmm(attn, v)

        gradient_hook = attn.register_hook(self.save_attn_gradient) # modified
        # print(attn_gradients)

        # print(attn_gradients, 'attn_gradients')
        # print(attn.grad, ' attn.grad')
        # print(attn_gradients, 'attn_gradients')

        return output, attn

    def save_attn_gradient(self, gradient): # modified
        gradient = gradient.view(self.bsz, self.num_heads, self.tgt_len, self.src_len)
        gradient = gradient.sum(dim=1) / self.num_heads
        self.attention_map_gradients = gradient

    def save_attn_map(self, attn):
        attn = attn.view(self.bsz, self.num_heads, self.tgt_len, self.src_len)
        attn = attn.sum(dim=1) / self.num_heads
        self.attention_map = attn

    def get_attn_map(self):
        return self.attention_map

    def get_attn_gradients(self):
        return self.attention_map_gradients


def _in_projection_packed(
    q: Tensor,
    k: Tensor,
    v: Tensor,
    w: Tensor,
    b: Optional[Tensor] = None,
) -> List[Tensor]:
    r"""
    Performs the in-projection step of the attention operation, using packed weights.
    Output is a triple containing projection tensors for query, key and value.

    Args:
        q, k, v: query, key and value tensors to be projected. For self-attention,
            these are typically the same tensor; for encoder-decoder attention,
            k and v are typically the same tensor. (We take advantage of these
            identities for performance if they are present.) Regardless, q, k and v
            must share a common embedding dimension; otherwise their shapes may vary.
        w: projection weights for q, k and v, packed into a single tensor. Weights
            are packed along dimension 0, in q, k, v order.
        b: optional projection biases for q, k and v, packed into a single tensor
            in q, k, v order.

    Shape:
        Inputs:
        - q: :math:`(..., E)` where E is the embedding dimension
        - k: :math:`(..., E)` where E is the embedding dimension
        - v: :math:`(..., E)` where E is the embedding dimension
        - w: :math:`(E * 3, E)` where E is the embedding dimension
        - b: :math:`E * 3` where E is the embedding dimension

        Output:
        - in output list :math:`[q', k', v']`, each output tensor will have the
            same shape as the corresponding input tensor.
    """
    E = q.size(-1)
    if k is v:
        if q is k:
            # self-attention
            return linear(q, w, b).chunk(3, dim=-1)
        else:
            # encoder-decoder attention
            w_q, w_kv = w.split([E, E * 2])
            if b is None:
                b_q = b_kv = None
            else:
                b_q, b_kv = b.split([E, E * 2])
            return (linear(q, w_q, b_q),) + linear(k, w_kv, b_kv).chunk(2, dim=-1)
    else:
        w_q, w_k, w_v = w.chunk(3)
        if b is None:
            b_q = b_k = b_v = None
        else:
            b_q, b_k, b_v = b.chunk(3)
        return linear(q, w_q, b_q), linear(k, w_k, b_k), linear(v, w_v, b_v)


def _in_projection(
    q: Tensor,
    k: Tensor,
    v: Tensor,
    w_q: Tensor,
    w_k: Tensor,
    w_v: Tensor,
    b_q: Optional[Tensor] = None,
    b_k: Optional[Tensor] = None,
    b_v: Optional[Tensor] = None,
) -> Tuple[Tensor, Tensor, Tensor]:
    r"""
    Performs the in-projection step of the attention operation. This is simply
    a triple of linear projections, with shape constraints on the weights which
    ensure embedding dimension uniformity in the projected outputs.
    Output is a triple containing projection tensors for query, key and value.

    Args:
        q, k, v: query, key and value tensors to be projected.
        w_q, w_k, w_v: weights for q, k and v, respectively.
        b_q, b_k, b_v: optional biases for q, k and v, respectively.

    Shape:
        Inputs:
        - q: :math:`(Qdims..., Eq)` where Eq is the query embedding dimension and Qdims are any
            number of leading dimensions.
        - k: :math:`(Kdims..., Ek)` where Ek is the key embedding dimension and Kdims are any
            number of leading dimensions.
        - v: :math:`(Vdims..., Ev)` where Ev is the value embedding dimension and Vdims are any
            number of leading dimensions.
        - w_q: :math:`(Eq, Eq)`
        - w_k: :math:`(Eq, Ek)`
        - w_v: :math:`(Eq, Ev)`
        - b_q: :math:`(Eq)`
        - b_k: :math:`(Eq)`
        - b_v: :math:`(Eq)`

        Output: in output triple :math:`(q', k', v')`,
         - q': :math:`[Qdims..., Eq]`
         - k': :math:`[Kdims..., Eq]`
         - v': :math:`[Vdims..., Eq]`

    """
    Eq, Ek, Ev = q.size(-1), k.size(-1), v.size(-1)
    assert w_q.shape == (Eq, Eq), f"expecting query weights shape of {(Eq, Eq)}, but got {w_q.shape}"
    assert w_k.shape == (Eq, Ek), f"expecting key weights shape of {(Eq, Ek)}, but got {w_k.shape}"
    assert w_v.shape == (Eq, Ev), f"expecting value weights shape of {(Eq, Ev)}, but got {w_v.shape}"
    assert b_q is None or b_q.shape == (Eq,), f"expecting query bias shape of {(Eq,)}, but got {b_q.shape}"
    assert b_k is None or b_k.shape == (Eq,), f"expecting key bias shape of {(Eq,)}, but got {b_k.shape}"
    assert b_v is None or b_v.shape == (Eq,), f"expecting value bias shape of {(Eq,)}, but got {b_v.shape}"
    return linear(q, w_q, b_q), linear(k, w_k, b_k), linear(v, w_v, b_v)

def linear(input: Tensor, weight: Tensor, bias: Optional[Tensor] = None) -> Tensor:
    r"""
    Applies a linear transformation to the incoming data: :math:`y = xA^T + b`.

    This operator supports :ref:`TensorFloat32<tf32_on_ampere>`.

    Shape:

        - Input: :math:`(N, *, in\_features)` N is the batch size, `*` means any number of
          additional dimensions
        - Weight: :math:`(out\_features, in\_features)`
        - Bias: :math:`(out\_features)`
        - Output: :math:`(N, *, out\_features)`
    """
    if has_torch_function_variadic(input, weight, bias):
        return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
    return torch._C._nn.linear(input, weight, bias)

[Pytorch] Sementation mask 시각화 하기

AI 꿈나무 — Sun, 26 Jun 2022 20:40:42 +0900

이미지를 segmentation 모델로 전달하여 pred를 얻었다고 가정하겠습니다.

for image, target in data_loader:
    pred_masks = model(image) # [N, H, W], dtype= Tensor.bool

이 pred_masks를 matplotlib를 사용하여 시각화 하겠습니다.

우선, pred_masks, target, image를 동일한 사이즈로 resize 해줘야 합니다.

안되어있는 경우 resize 합니다.

import torchvision.transforms.functional as TF

h, w = image.shape[2], image.shape[3]
pred_masks = TF.resize(pred_masks, (h, w)).type(torch.bool) # [N, H, W]
target = TF.resize(target, {h, w)).type(torch.bool) # [N, H, W]

시각화 함수를 정의합니다.

제 경우에는 sentence와 iou도 함께 존재하는데요. 이 값까지 함께 시각화 했습니다.

연구에 필요한 코드를 짜고나서 블로그에 업로드하기 때문에, 보시는 분들은 불필요한 코드가 많다고 느끼실 수 있습니다.

def show_result(image, target, preds, sentences, ious, args, height, width, opacity=0.5):
    img = image.squeeze().cpu().numpy().transpose(1,2,0) # [3, 480, 480] -> [480, 480, 3]
    target = target.squeeze().cpu().numpy().astype(bool) # [1, H, W] -> [H, W]
    ious = [iou.cpu().numpy() for iou in ious]
    preds = [pred.squeeze(0).cpu().numpy().astype(bool) for pred in preds]
    

    mean = ([0.485, 0.456, 0.406])
    std = ([0.229, 0.224, 0.225])

    # re-nomalize
    for c, (mean_c, std_c) in enumerate(zip(mean, std)):
        img[:,:,c] *= std_c
        img[:,:,c] += mean_c

    color = np.array(ImageColor.getrgb('red'), dtype=np.uint8) / 255# tuple

    gt = copy.copy(img)
    # apply mask
    for c in range(3):
        gt[:,:,c] = np.where(target == 1,
                             gt[:,:,c] * opacity + (1 - opacity) * color[c],
                             gt[:,:,c])


    fig = plt.figure(figsize=(30,10),constrained_layout=True)
    specs = gridspec.GridSpec(nrows=2, ncols= len(sentences) + 1)
    ax1 = fig.add_subplot(specs[0,0])
    ax1.set_title(f'mean_iou: {np.mean(ious):.2f} \n Image', fontsize=25)
    ax1.axis('off')
    ax1.imshow(img)

    ax2 = fig.add_subplot(specs[0,1])
    ax2.set_title('GT', fontsize=20)
    ax2.axis('off')
    ax2.imshow(gt)

    count = 0
    for pred, sentence, iou in zip(preds, sentences, ious):
        mask_seg = copy.copy(img)

        for c in range(3):
            mask_seg[:,:,c] = np.where(pred == 1,
                                       mask_seg[:,:,c] * opacity + (1 - opacity) * color[c],
                                       mask_seg[:,:,c])

        ax = fig.add_subplot(specs[1, count])
        ax.set_title(f'IoU: {iou:.2f} \n {sentence}', fontsize=25)
        ax.axis('off')
        ax.imshow(mask_seg)

        count += 1

    os.makedirs(f'./show_result/{args.dataset}_{args.split}_{args.clip_model}_{height}x{width}/', exist_ok=True)
    show_dir = f'./show_result/{args.dataset}_{args.split}_{args.clip_model}_{height}x{width}/M{np.mean(ious):.2f}_H{np.max(ious):.2f}_L{np.min(ious):.2f}_{sentence}.jpg'


    plt.savefig(show_dir)
    plt.show()

show_result(imgs, target, result_segs, sentence_raw, this_ious, args, Height, Width)

파이참에서 원격 주피터 노트북 사용하기

AI 꿈나무 — Sun, 26 Jun 2022 16:30:26 +0900

파이참으로 원격 서버의 interpreter를 사용하여 주피터 노트북 사용하는 방법을 알아보겠습니다.

우선 local에 ipynb 파일을 생성합니다.

ipynb 파일을 실행하면 상단에 interpreter를 설정할 수 있는 옵션이 있습니다.

다음 옵션으로 interpreter를 설정하면

Jupyter server process failed to start Illegal char :> at index 4

에러가 발생합니다.

configured server 로 설정을 해줘야 하는데요.

원격 서버에서 jupyter notebook을 실행 후 생성된 주소와 토큰을 configured server에 입력해주면 됩니다.

127.0.0.1 부분을 원격 서버의 ip로 설정해주어야 합니다.

[Python] list()와 []의 차이점

AI 꿈나무 — Thu, 16 Jun 2022 01:55:25 +0900

numpy.array 타입을 가진 변수 a가 있다고 가정하겠습니다.
type(a) = np.array

list(a)는 np.array type을 지닌 a를 list type으로 변경합니다

[a]는 list[np.array] 처럼 np.array를 list로 감쌉니다.

Selenium을 사용해서 백준 유저가 푼 문제 크롤링 하기

AI 꿈나무 — Wed, 15 Jun 2022 23:35:27 +0900

리눅스 환경에서 진행했습니다.

아래 명령어를 실행하여 크롬 드라이버를 설치합니다.

apt install chromium-chromedriver

백준 사이트에서 유저가 푼 문제를 받아오는 코드입니다.

def get_solved_problem(name='chleodnr3'):
    options = webdriver.ChromeOptions()
    options.add_argument("headless")

    # apt install chromium-chromedriver

    name = 'chleodnr3'
    driver = webdriver.Chrome(options=options)
    driver.get("https://www.acmicpc.net/user/{}".format(name))
    element = driver.find_element(By.CLASS_NAME, 'problem-list')

    problems = element.text
    problems = problems.split(' ')
    problems = list(map(int, problems))
    return problems

아래 함수를 호출하면 유저가 푼 문제를 list[int] 형태로 반환합니다.

파이참 한국어에서 영어로 설정하기

AI 꿈나무 — Sun, 12 Jun 2022 17:52:00 +0900

파이참 버전 업데이트를 하니 자동으로 한국어팩이 설치되어 한국어 파이참이 적용되었다.

영어 파이참을 오래 써왔던 터라 오히려 한국어 버전이 불편했는데 이번 기회에 한국어에서 영어로 변경해보고 방법까지 공유하겠다.

setting에서 plugins에 들어간다.

위에 있는 installed(설치됨) 탭에 들어가면 한국어 언어팩을 볼 수 있는데, 체크를 풀어주고 apply를 누르면 된다.

파이참 버전 업데이트

AI 꿈나무 — Sun, 12 Jun 2022 17:49:42 +0900

파이참 버전 업데이트 하는 방법을 알아보자.

상위 도구 바에서 Help를 누른 후 Check for Updates를 선택하면 된다.

21년도 버전을 사용하다가, 22년도 버전을 사용하려는 이유는 원격 서버에 있는 .py 파일을 간단하게 새로고침 할 수 있는 기능이 있기 때문이다 ㅎㅎ

[Pytorch] List를 Tensor로 변경하기. torch.stack

AI 꿈나무 — Sun, 22 May 2022 23:14:10 +0900

list에 tensor가 담겨져 있는 경우, list를 없애고 torch.tensor로 바꾸는 방법은 아래와 같다.

# segm_result = list[tensor]
segm_result = torch.stack(segm_result, dim=0)

[PyTorch] 학습 progress bar 설정하기

AI 꿈나무 — Tue, 17 May 2022 23:14:17 +0900

학습시에 진행과정, 경과시간, 예상시간을 확인하고 싶은 경우, 사용가능한 함수가 mmcv에 구현되어 있다.

import mmcv

prog_bar = mmcv.ProgressBar(len(dataset))

for i, data in enumerate(data_loader):
	...
    
    for _ in range(batch_size):
    	prog_bar.update()

[논문 읽기] Open-World Entity Segmentation(2021)

AI 꿈나무 — Fri, 4 Mar 2022 16:34:03 +0900

Open-World Entity Segmentation

https://arxiv.org/abs/2107.14228

Open-World Entity Segmentation

We introduce a new image segmentation task, called Entity Segmentation (ES), which aims to segment all visual entities (objects and stuffs) in an image without predicting their semantic labels. By removing the need of class label prediction, the models tra

arxiv.org

class agnostic 하게 segmentation 하는 모델이다. entity를 segment 하는데, entity는 stuff와 thing 모두를 포함한다. class-agnostic한 segmentation을 하므로 class prediction에 신경쓸 필요가 없다. 따라서 class-specific하게 학습된 모델보다 localization 능력이 향상된다.

FCOS의 center-based representation과 CondInst의 generate kernel 방식을 사용한다.

FPN에 부착된 head는 kernel, entityness, centerness, locaization 을 출력한다. localization은 바운딩박스를 생성하기 위함이고 추후에 NMS를 사용하여 output을 정제하기 위함이다. entityness와 centerness는 probability map이다.

논문에서 제안하는 방식은 global kernel bank를 사용하여 conv weight 를 생성한다.

entity들이 공통적으로 지니고 있는 property와 entity 사이에 서로 다른 property를 추출하기 위해 dynamic weight와 static weight를 생성한다. 생성된 weight들을 순열방식으로 7개의 path를 만들어서 training에 활용.

inference는 dynamic weight로만 이루어진 7번 path만을 사용.

또 overlab suppression 방법을 제안한다. 모델이 예측한 mask가 overlap 될 수 있는데 이를 억제하는 역할이다.

7번 path에 존재하는 3개의 weight를 mask feature map에 적용하여 평균낸 값을 사용한다.

또 학습시에도 loss를 가한다.

[논문 읽기] CondInst(2020), Conditional Convolutions for Instance Segmentation

AI 꿈나무 — Fri, 4 Mar 2022 16:10:09 +0900

Conditional Convolutions for Instance Segmentation

https://arxiv.org/abs/2003.05664

Conditional Convolutions for Instance Segmentation

We propose a simple yet effective instance segmentation framework, termed CondInst (conditional convolutions for instance segmentation). Top-performing instance segmentation methods such as Mask R-CNN rely on ROI operations (typically ROIPool or ROIAlign)

arxiv.org

FPN으로 추출한 feature map에 1x1 conv weight, bias를 생성하는 head를 부착한다.

생성한 weight는 pixel wise하게 생성되므로 instance aware한 정보를 지니고 있다.

이 weight를 Mask head로 사용하여 mask를 추출한다.

[논문 읽기] CaSP, Class agnostic Semi-Supervised Pretraining for Detection and Segmentation

AI 꿈나무 — Wed, 2 Mar 2022 16:37:33 +0900

CaSP, Class agnostic Semi-Supervised Pretraining for Detection and Segmentation

https://arxiv.org/abs/2112.04966

CaSP: Class-agnostic Semi-Supervised Pretraining for Detection and Segmentation

To improve instance-level detection/segmentation performance, existing self-supervised and semi-supervised methods extract either very task-unrelated or very task-specific training signals from unlabeled data. We argue that these two approaches, at the two

arxiv.org

semi seg or detection 논문.

기존의 semi 는 labeled data와 unlabeled data를 joint하여 학습시킨다.

이 논문은 class agnoistic unlabeled data로 pre-training 한뒤에 labeled data로 fine-tuning을 한다.

위 논문은 3가지 process가 존재한다.

(1) Early pretraining

detection dataset에서 class 정보를 제거하고 class-agnoistic한 data로 detector를 학습시킨다. localization만 집중하기 때문에 localization 성능이 class-specific data로 학습 시킨 것 보다 agnoistic 한게 더 높게 나온다.

(2) Late pretraining

(1)로 학습한 모델로 unlabeled data로부터 pseudo label을 생성해 target detector를 학습시킨다. 역시 class agnoistic하게.

(3) fine-tuning

이제 target detector를 down-stream dataset에 class 정보와 함께 학습시킨다.

GitHub - Seonghoon-Yu/AI_Paper_Review: 까먹으면 다시 보려고 정리합니다.

까먹으면 다시 보려고 정리합니다. Contribute to Seonghoon-Yu/AI_Paper_Review development by creating an account on GitHub.

github.com

[python] 파이썬 패키지, 모듈 저장 위치 확인하기

AI 꿈나무 — Fri, 25 Feb 2022 03:34:00 +0900

import clip
import inspect

print(inspect.getfile(clip))