AWS EC2에 pgvector 설치하고 접속하기

AWS EC2에 pgvector 설치하고 접속하기

AWS EC2에 PostgreSQL을 설치하고 접속해보겠다.
사실 AWS에는 관계형 데이터베이스를 편하게 다룰 수 있는 서비스인 RDS를 제공하기는 하지만 과금이 많이 되는 경향도 있고 여러 이유로 PostgreSQL을 직접 EC2에 설치해서 사용하기로 했다.

1. postgresql 설치

EC2에 접근하여 postgresql(이하 PG)를 설치한다.

1
2
sudo amazon-linux-extras install postgresql10 epel -y
sudo yum install postgresql-server postgresql-devel -y

에러 발생

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00:00
amzn2extra-docker | 2.9 kB 00:00:00
amzn2extra-epel | 3.0 kB 00:00:00
amzn2extra-postgresql10 | 2.9 kB 00:00:00


One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:

1. Contact the upstream for the repository and get them to fix the problem.

2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).

3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...

4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:

yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>

5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:

yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again

EC2가 yum 서버와 통신을 못해서 문제가 발생한 것이다.
즉, EC2가 외 인터넷과 통신이 되지 않는 것이다.

이러한 경우 인터넷 게이트웨이를 생성하여 VPC및 서브넷과 연동해주면 된다.

참고 : https://garve32.tistory.com/63

AWS EC2 인스턴스에 파일 업로드 및 다운로드

파일을 옮기기 위해 퍼블릭 IPv4 고정 IP 설정을 해준다.

https://rypro.tistory.com/227

S3에 업로드한 pgvector 파일이용하여 설치하기

S3와 EC2를 연동하는 방법은 다른 포스팅을 참고바란다.

1
2
3
4
5
6
7
8
[root@ec2-ct01-dev-slm-app-03 ~]# aws s3 cp s3://aipin-bucket/pgvector.zip /root
download: s3://aipin-bucket/pgvector.zip to ./pgvector.zip

[root@ec2-ct01-dev-slm-app-03 ~]# unzip /root/pgvector.zip -d /root
Archive: /root/pgvector.zip
inflating: /root/README.md
inflating: /root/docker-compose.yml
extracting: /root/init_db.sql

Docker 실행

1
2
$ sudo systemctl start docker
$ sudo systemctl enable docker

1. amzn2extra 리포지토리 활성화

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
$ sudo amazon-linux-extras enable postgresql10

2 httpd_modules available [ =1.0 =stable ]
3 memcached1.5 available \
[ =1.5.1 =1.5.16 =1.5.17 ]
6 *postgresql10=latest enabled [ =10 =stable ]
9 R3.4 available [ =3.4.3 =stable ]
10 rust1 available \
[ =1.22.1 =1.26.0 =1.26.1 =1.27.2 =1.31.0 =1.38.0
=stable ]
18 libreoffice available \
[ =5.0.6.2_15 =5.3.6.1 =stable ]
19 gimp available [ =2.8.22 ]
20 †docker=latest enabled \
[ =17.12.1 =18.03.1 =18.06.1 =18.09.9 =stable ]
21 mate-desktop1.x available \
[ =1.19.0 =1.20.0 =stable ]
22 GraphicsMagick1.3 available \
[ =1.3.29 =1.3.32 =1.3.34 =stable ]
24 epel=latest enabled [ =7.11 =stable ]
25 testing available [ =1.0 =stable ]
26 ecs available [ =stable ]
27 †corretto8 available \
[ =1.8.0_192 =1.8.0_202 =1.8.0_212 =1.8.0_222 =1.8.0_232
=1.8.0_242 =stable ]
32 lustre2.10 available \
[ =2.10.5 =2.10.8 =stable ]
33 †java-openjdk11 available [ =11 =stable ]
34 lynis available [ =stable ]
36 BCC available [ =0.x =stable ]
37 mono available [ =5.x =stable ]
38 nginx1 available [ =stable ]
40 mock available [ =stable ]
43 livepatch available [ =stable ]
44 †python3.8 available [ =stable ]
45 haproxy2 available [ =stable ]
46 collectd available [ =stable ]
47 aws-nitro-enclaves-cli available [ =stable ]
48 R4 available [ =stable ]
49 kernel-5.4 available [ =stable ]
50 selinux-ng available [ =stable ]
52 tomcat9 available [ =stable ]
53 unbound1.13 available [ =stable ]
54 †mariadb10.5 available [ =stable ]
55 kernel-5.10 available [ =stable ]
56 redis6 available [ =stable ]
58 †postgresql12 available [ =stable ]
59 †postgresql13 available [ =stable ]
60 mock2 available [ =stable ]
61 dnsmasq2.85 available [ =stable ]
62 kernel-5.15 available [ =stable ]
63 †postgresql14 available [ =stable ]
64 firefox available [ =stable ]
65 lustre available [ =stable ]
66 †php8.1 available [ =stable ]
67 awscli1 available [ =stable ]
68 †php8.2 available [ =stable ]
69 dnsmasq available [ =stable ]
70 unbound1.17 available [ =stable ]
72 collectd-python3 available [ =stable ]
* Extra topic has reached end of support.
† Note on end-of-support. Use 'info' subcommand.

Now you can install:
# yum clean metadata
# yum install postgresql

2. PostgreSQL 설치

1
2
3
4
5
6
7
8
9
10
$ sudo yum --disablerepo=epel install postgresql10 postgresql10-server
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00:00
amzn2extra-docker | 2.9 kB 00:00:00
amzn2extra-epel | 3.0 kB 00:00:00
amzn2extra-postgresql10 | 2.9 kB 00:00:00
No package postgresql10 available.

No package postgresql10-server available.
Error: Nothing to do

에러가 났다.

amazon-linux-extras 리포지토리 확인

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
$ sudo amazon-linux-extras list

# 출력에서 postgresql10이 활성화되어 있는지 확인합니다. 만약 활성화되어 있지 않다면, 다시 활성화.
$ sudo amazon-linux-extras enable postgresql10

# 리포지토리 캐시 갱신
$ sudo yum clean all

# 다시 설치
$ sudo yum --disablerepo=epel install postgresql10 postgresql10-server
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00:00
amzn2extra-docker | 2.9 kB 00:00:00
amzn2extra-epel | 3.0 kB 00:00:00
amzn2extra-postgresql10 | 2.9 kB 00:00:00
(1/9): amzn2-core/2/x86_64/group_gz | 2.7 kB 00:00:00
(2/9): amzn2-core/2/x86_64/updateinfo | 935 kB 00:00:00
(3/9): amzn2extra-epel/2/x86_64/primary_db | 1.8 kB 00:00:00
(4/9): amzn2extra-postgresql10/2/x86_64/updateinfo | 55 B 00:00:00
(5/9): amzn2extra-postgresql10/2/x86_64/primary | 15 kB 00:00:00
(6/9): amzn2extra-docker/2/x86_64/primary_db | 102 kB 00:00:00
(7/9): amzn2extra-epel/2/x86_64/updateinfo | 76 B 00:00:00
(8/9): amzn2extra-docker/2/x86_64/updateinfo | 16 kB 00:00:00
(9/9): amzn2-core/2/x86_64/primary_db | 68 MB 00:00:00
No package postgresql10 available.
No package postgresql10-server available.
Error: Nothing to do

계속 epel이 문제인듯 하니 epel enabled = 0을 바꿔서 비활성화 시켜버림

1
2
3
4
5
6
7
8
9
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
metalink=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

yum cache 갱신

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ sudo yum clean all
$ sudo yum makecache
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core | 3.6 kB 00:00:00
amzn2extra-docker | 2.9 kB 00:00:00
amzn2extra-epel | 3.0 kB 00:00:00
amzn2extra-postgresql10 | 2.9 kB 00:00:00
(1/17): amzn2-core/2/x86_64/group_gz | 2.7 kB 00:00:00
(2/17): amzn2-core/2/x86_64/updateinfo | 935 kB 00:00:00
(3/17): amzn2-core/2/x86_64/filelists_db | 61 MB 00:00:00
(4/17): amzn2-core/2/x86_64/primary_db | 68 MB 00:00:00
(5/17): amzn2extra-docker/2/x86_64/filelists_db | 34 kB 00:00:00
(6/17): amzn2extra-docker/2/x86_64/updateinfo | 16 kB 00:00:00
(7/17): amzn2extra-docker/2/x86_64/primary_db | 102 kB 00:00:00
(8/17): amzn2extra-docker/2/x86_64/other_db | 32 kB 00:00:00
(9/17): amzn2-core/2/x86_64/other_db | 21 MB 00:00:00
(10/17): amzn2extra-epel/2/x86_64/updateinfo | 76 B 00:00:00
(11/17): amzn2extra-epel/2/x86_64/filelists_db | 882 B 00:00:00
(12/17): amzn2extra-epel/2/x86_64/primary_db | 1.8 kB 00:00:00
(13/17): amzn2extra-epel/2/x86_64/other_db | 507 B 00:00:00
(14/17): amzn2extra-postgresql10/2/x86_64/updateinfo | 55 B 00:00:00
(15/17): amzn2extra-postgresql10/2/x86_64/primary | 15 kB 00:00:00
(16/17): amzn2extra-postgresql10/2/x86_64/filelists | 121 kB 00:00:00
(17/17): amzn2extra-postgresql10/2/x86_64/other | 6.2 kB 00:00:00
Metadata Cache Created

다시 설치

1
2
3
4
$ sudo yum install postgresql10 postgresql10-server
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
No package postgresql10 available.
No package postgresql1

여전히 안됨

명령어가 틀린듯

1
2
3
4
5
6
7
yum install postgresql
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2extra-docker | 2.9 kB 00:00:00
amzn2extra-epel | 3.0 kB 00:00:00
amzn2extra-postgresql10 | 2.9 kB 00:00:00
Package postgresql-10.21-1.amzn2.0.1.x86_64 already installed and latest version
Nothing to do

3. PostgreSQL 데이터베이스 초기화

1
2
3
$ sudo /usr/pgsql-10/bin/postgresql-10-setup initdb

# 안먹음

계속 psql에 문제가 있어서 PostgreSQL을 설치하기 위해 필요한 RPM 파일을 인터넷이 되는 외부 시스템에서 다운로드한 후, 폐쇄망 서버로 전송하여 설치하는 방법으로 진행

docker-compose 설치

1
2
3
4
5
6
7
8
9
10
$ curl -L "https://github.com/docker/compose/releases/download/v2.20.2/docker-compose-$(uname -s)-$(uname -m)" -o docker-compose

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 57.9M 100 57.9M 0 0 14.5M 0 0:00:03 0:00:03 --:--:-- 21.4M

# 옮기기
$ aws s3 cp s3://aipin-bucket/docker-compose /root
download: s3://aipin-bucket/docker-compose to ./docker-compose

Docker Compose 파일을 적절한 위치로 이동시키고 실행 권한을 부여

1
2
$ sudo mv docker-compose /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

잘 설치되었는지 확인

1
2
$ docker-compose --version
Docker Compose version v2.20.2

docker-compose로 pgvector 올리기

1
2
$ docker-compose up -d
[+] Running 1/1

postgre를 다시 다른 방법으로 설치해보자

1
2
3
4
5
6
7
8
9
$ aws s3 cp s3://aipin-bucket/postgresql-9.6.16.tar.gz /root
download: s3://aipin-bucket/postgresql-9.6.16.tar.gz to ../postgresql-9.6.16.tar.gz

# 패키지 파일 압축해제
$ tar -zxvf postgresql-9.6.16.tar.gz
# 설치 디렉토리는 디폴트로 함, --prefix=/usr/local/pgsql 이렇게 하는 경우도 있는듯
$ ./configure --without-readline
$ make && make check
$ make install

PostgreSQL 사용자 및 디렉토리 설정

1
2
3
4
5
sudo useradd postgres
sudo mkdir /usr/local/pgsql/data
sudo chown postgres /usr/local/pgsql/data
sudo mkdir /usr/local/pgsql/logs
sudo chown postgres /usr/local/pgsql/logs

데이터베이스 초기화

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
su - postgres
[postgres@ec2-ct01-dev-slm-app-03 ~]$ /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /usr/local/pgsql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Asia/Seoul
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

/usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l logfile start

PostgreSQL 서버 시작

PostgreSQL 서버를 시작합니다. PostgreSQL 사용자로 전환하여 PostgreSQL 서버를 백그라운드에서 실행

1
2
[postgres@ec2-ct01-dev-slm-app-03 ~]$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/logs/logfile start
server starting

PostgreSQL 서버 자동 시작 설정 (Optional)

시스템 부팅 시 PostgreSQL 서버가 자동으로 시작되도록 설정할 수 있습니다. /etc/systemd/system/postgresql.service 파일을 생성하고 다음 내용을 추가합니다.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
sudo tee /etc/systemd/system/postgresql.service <<EOF
[Unit]
Description=PostgreSQL database server
After=network.target

[Service]
Type=forking
User=postgres
ExecStart=/usr/local/pgsql/bin/pg_ctl start -D /usr/local/pgsql/data -l /usr/local/pgsql/logs/logfile
ExecStop=/usr/local/pgsql/bin/pg_ctl stop -D /usr/local/pgsql/data
ExecReload=/usr/local/pgsql/bin/pg_ctl reload -D /usr/local/pgsql/data

[Install]
WantedBy=multi-user.target
EOF

PostgreSQL 접속

1
2
3
4
5
6
7
8
9
10
11
$ /usr/local/pgsql/bin/psql
psql (9.6.16)
Type "help" for help.

# 비번 설정
postgres=#
postgres=# ALTER USER postgres WITH PASSWORD 'new1234';
ALTER ROLE

# 종료
postgres=# \q

잘 돌고있는지 확인

1
2
3
4
5
6
7
8
9
10
11
ps aux | grep postgres
root 16797 0.0 0.0 200768 3912 pts/1 S 22:04 0:00 su - postgres
postgres 16798 0.0 0.1 124744 4012 pts/1 S 22:04 0:00 -bash
postgres 17004 0.0 0.4 277808 16760 pts/1 S 22:06 0:00 /usr/local/pgsql/bin/postgres -D /usr/local/pgsql/data
postgres 17009 0.0 0.0 277808 2748 ? Ss 22:06 0:00 postgres: checkpointer process
postgres 17010 0.0 0.0 277808 2748 ? Ss 22:06 0:00 postgres: writer process
postgres 17011 0.0 0.0 277808 2748 ? Ss 22:06 0:00 postgres: wal writer process
postgres 17012 0.0 0.1 278208 5504 ? Ss 22:06 0:00 postgres: autovacuum launcher process
postgres 17013 0.0 0.0 132824 2300 ? Ss 22:06 0:00 postgres: stats collector process
postgres 17840 0.0 0.1 162296 3960 pts/1 R+ 22:14 0:00 ps aux
postgres 17844 0.0 0.0 119420 912 pts/1 S+ 22:14 0:00 grep --color=auto postgres
1
2
3
4
5
6
7
8
9
10
11
12
[postgres@ec2-ct01-dev-slm-app-03 ~]$ /usr/local/pgsql/bin/psql
psql (9.6.16)
Type "help" for help.

postgres=# SELECT version();
version

----------------------------------------------------------------------------------------------------
-------
PostgreSQL 9.6.16 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 7.3.1 20180712 (Red Hat 7.3.1-17),
64-bit
(1 row)

다시 앞의 docker-compose로 pgvector 올리기

1
2
3
4
$ docker-compose up -d
[+] Running 1/1
✘ db Error 15.0s
Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

흐음…

postgres 종료

$ sudo su - postgres
Last login: Mon Jun 24 22:16:42 KST 2024 on pts/0
$ /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l /usr/local/pgsql/logs/logfile stop
waiting for server to shut down.... done
server stopped

- references
https://inblog.ai/guri-tech-blog/ec2%EC%97%90-postgressql-96-%EC%84%A4%EC%B9%98-%ED%8C%A8%ED%82%A4%EC%A7%80%EC%84%A4%EC%B9%98-tar-7393
https://wldnjd2.tistory.com/m/95

AWS EC2에 pgvector 설치하고 접속하기

https://hamin7.github.io/2024/06/17/AWS-Install-Postgresql-On-EC2/

Author

hamin

Posted on

2024-06-17

Updated on

2024-06-24

Licensed under

You need to set install_url to use ShareThis. Please set it in _config.yml.
You forgot to set the business or currency_code for Paypal. Please set it in _config.yml.

Comments

You forgot to set the shortname for Disqus. Please set it in _config.yml.
You need to set client_id and slot_id to show this AD unit. Please set it in _config.yml.