AWS EC2 배포 중 Cannot allocate memory (12) 에러

AWS EC2에 배포 중 아래와 같은 에러가 났다.

2024-09-11 06:04:15,674 - DEBUG - Attempting to release lock 140279398944864 on /root/.cache/huggingface/hub/.locks/models--Dongjin-kr--ko-reranker/002b874aa6e3b367da62d14acba828ca8d1f4bf50830747ef922d45bf71daaa2.lock
2024-09-11 06:04:15,674 - DEBUG - Lock 140279398944864 released on /root/.cache/huggingface/hub/.locks/models--Dongjin-kr--ko-reranker/002b874aa6e3b367da62d14acba828ca8d1f4bf50830747ef922d45bf71daaa2.lock
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: read).
Your token has been saved to /root/.cache/huggingface/token
Login successful
The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: read).
Your token has been saved to /root/.cache/huggingface/token
Login successful
Traceback (most recent call last):
  File "/app/src/main.py", line 3, in <module>
    import utils as utils
  File "/app/src/utils.py", line 5, in <module>
    import controller.rag_retriever_plugin_controller as rag_retriever_plugin_controller
  File "/app/src/controller/rag_retriever_plugin_controller.py", line 13, in <module>
    plugin = RagRetrieverPlugin(documents)
  File "/app/src/plugins/rag_retriever_plugin.py", line 30, in __init__
    super().__init__(name, model, retriever, rerank_model)
  File "/usr/local/lib/python3.10/site-packages/aipin/Plugin/providers/plugin_retriever.py", line 21, in __init__
    self.rerank = Rerank(rerank_model)
  File "/usr/local/lib/python3.10/site-packages/aipin/data/retriever.py", line 66, in __init__
    model = AutoModelForSequenceClassification.from_pretrained(model_path)
  File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3735, in from_pretrained
    with safe_open(resolved_archive_file, framework="pt") as f:
RuntimeError: unable to mmap 2239614572 bytes from file </root/.cache/huggingface/hub/models--Dongjin-kr--ko-reranker/snapshots/5f3bb02f3baa05dea3e3c6653ada191f2cc20d91/model.safetensors>: Cannot allocate memory (12)
Traceback (most recent call last):
  File "/app/src/main.py", line 3, in <module>
    import utils as utils
  File "/app/src/utils.py", line 5, in <module>
    import controller.rag_retriever_plugin_controller as rag_retriever_plugin_controller
  File "/app/src/controller/rag_retriever_plugin_controller.py", line 13, in <module>
    plugin = RagRetrieverPlugin(documents)
  File "/app/src/plugins/rag_retriever_plugin.py", line 30, in __init__
    super().__init__(name, model, retriever, rerank_model)
  File "/usr/local/lib/python3.10/site-packages/aipin/Plugin/providers/plugin_retriever.py", line 21, in __init__
    self.rerank = Rerank(rerank_model)
  File "/usr/local/lib/python3.10/site-packages/aipin/data/retriever.py", line 66, in __init__
    model = AutoModelForSequenceClassification.from_pretrained(model_path)
  File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3735, in from_pretrained
    with safe_open(resolved_archive_file, framework="pt") as f:

이 오류는 Hugging Face에서 큰 모델 파일을 로드하려고 할 때 시스템 메모리가 부족해서 발생한 것이다.
unable to mmap 2239614572 bytes from file 오류는 메모리 부족을 의미하며, “Cannot allocate memory (12)”라는 메시지가 이를 명확히 보여준다.

1. 메모리 사용량 줄이기

• 스왑 공간 증가: 시스템에 RAM이 부족하다면 스왑 공간을 늘려보는 것이 좋다.
• 스왑 공간은 메모리가 부족할 때 디스크를 임시 메모리로 사용하는 방법이다. 성능은 다소 느려질 수 있지만 메모리 문제를 해결하는 데 도움이 될 수 있다.
• Linux 시스템에서 스왑 공간을 생성하고 활성화하는 방법

  [root@ip-10-71-176-76 ~]# fallocate -l 4G /swapfile
  [root@ip-10-71-176-76 ~]# sudo chmod 600 /swapfile
  [root@ip-10-71-176-76 ~]# sudo mkswap /swapfile
  Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
  no label, UUID=0ec98152-ae9d-4bf9-ab06-b697b5c99799
  [root@ip-10-71-176-76 ~]# sudo swapon /swapfile