# 消息队列pulsar

1. ZooKeeper： ZooKeeper是一个分布式协调服务，用于管理Pulsar集群的元数据和协调Pulsar Broker之间的通信。在搭建Pulsar集群之前，你需要先搭建和配置一个ZooKeeper集群。
2. Pulsar Bookkeeper： Pulsar Bookkeeper是Pulsar的持久化存储组件，用于可靠地存储消息和元数据。你需要搭建和配置一个Pulsar Bookkeeper集群，并将其连接到ZooKeeper集群。Pulsar Broker将使用Bookkeeper来持久化消息。
3. Pulsar Broker： Pulsar Broker是Pulsar的核心组件，负责接收、存储和分发消息。你需要搭建和配置一个或多个Pulsar Broker节点，并将它们连接到ZooKeeper集群和Pulsar Bookkeeper集群。Pulsar Broker将使用ZooKeeper来管理元数据和协调Broker之间的通信。

在配置Pulsar Broker时，需要指定ZooKeeper的地址和端口，以及Pulsar Bookkeeper的地址和端口。Pulsar Broker将使用这些信息来连接到ZooKeeper和Pulsar Bookkeeper，以获取元数据和存储消息。

一旦完成了ZooKeeper、Pulsar Bookkeeper和Pulsar Broker的搭建和配置，你就可以使用Pulsar Client来发送和接收消息了。Pulsar Client将连接到Pulsar Broker，并通过Pulsar Bookkeeper来读写消息。

### 创建Overlay网络

在任一管理节点执行如下命令创建名为 `myapp` 的 overlay 网络：

```
docker network create -d overlay  --attachable  myapp
```

> ​	`--attachable` 声明当前创建的网络为：其他节点可以访问的网络。

网络创建完成后，在任一管理节点执行 `docker network ls` 可以看到创建完成的 `myapp` 网络：

```
NETWORK ID     NAME              DRIVER    SCOPE
4a72bbeffbb8   bridge            bridge    local
4745b6b1d9f9   docker_gwbridge   bridge    local
477401a250c1   host              host      local
tgureb3sg8oz   ingress           overlay   swarm
2kekj9ksfrnx   myapp             overlay   swarm
```

#### 创建目录 

为便于数据管理，切换到 pulsar 用户创建数据目录

```
mkdir -p /home/pulsar/zookeeper/{data,log} /home/pulsar/bookkeeper/{journal,ledgers}
```

可以通过docker stack deploy -c 来搭建消息队列集群

## [Docker](https://www.xinbaoku.com/docker/) Stack 

[Docker](https://www.xinbaoku.com/docker/) Stack 为解决该问题而生，Docker Stack 通过提供期望状态、滚动升级、简单易用、扩缩容、健康检查等特性简化了应用的管理，这些功能都封装在一个完美的声明式模型当中

如果了解Docker Compose，就会发现Docker Stack非常简单。事实上在许多方面，Stack一直是期望的Compose——完全集成到Docker中，并能够管理应用的整个生命周期

从体系结构上来讲，Stack位于Docker应用层级的最顶端。Stack基于服务进行构建，而服务又基于容器

[Docker Stack命令 (xinbaoku.com)](https://www.xinbaoku.com/archive/Bea4Fqc6.html)

![image-20230725164830072](https://ye5201314-1312898079.cos.ap-guangzhou.myqcloud.com/image-20230725164830072.png)

```
常用命令：
docker stsack deploy
用于根据 Stack 文件（通常是 docker-stack.yml）部署和更新 Stack 服务的命令。
docker stack ls
会列出 Swarm 集群中的全部 Stack，包括每个 Stack 拥有多少服务。
docker stack ps
列出某个已经部署的 Stack 相关详情。该命令支持 Stack 名称作为其主要参数，列举了服务副本在节点的分布情况，以及期望状态和当前状态。
docker stack rm
命令用于从 Swarm 集群中移除 Stack。移除操作执行前并不会进行二次确认。
docker stack services <stack_name>
查看<service_id>
docker service logs <service_id>
查看日志
```

#### 部署准备

以下操作都在主节点运行

将pulsar-docker.tar包放到虚拟机上，并解压

```
[root@node-01 pulsar]# ll
total 28
drwxr-xr-x. 2 root root     6 Jul 20 02:47 admin
drwxr-xr-x. 2 root root    32 Jul 25 03:50 bookkeeper
drwxr-xr-x. 2 root root    32 Jul 25 03:52 broker
-rw-r--r--. 1 root root 12737 Jul 19 04:14 docker-compose.tmp
-rw-r--r--. 1 root root  4769 Jul 19 22:32 docker-compose.yml
drwxr-xr-x. 2 root root    32 Jul 20 06:28 proxy
-rw-r--r--. 1 root root  2033 Jul 20 21:32 pulsar.tar.gz
drwxr-xr-x. 2 root root    32 Jul 25 02:48 zookeeper
```

## 部署zookeeper(调度)

ZooKeeper是一个分布式的开源协调服务，用于管理和协调分布式系统中的各种任务和配置信息。它提供了一个简单的接口，可以让开发人员构建可靠的分布式应用程序。

1. ZooKeeper是一个分布式的开源协调服务，用于管理和协调分布式系统中的各种任务和配置信息。它提供了分布式协调原语、数据管理、一致性和可靠性、高性能和可扩展性、可靠的顺序性等特性，使得开发人员可以构建可靠和高效的分布式应用程序。

进入的zookeeper目录(/opt/目录下)（/data/目录下只存数据）

![image-20230725165401658](https://ye5201314-1312898079.cos.ap-guangzhou.myqcloud.com/image-20230725165401658.png)

编辑docker-compose.yml

```
将constraints: [ node.hostname == node1 ]/constraints: [ node.hostname == node2 ]/constraints: [ node.hostname == node3 ]改为相对应的主机名（所部署的虚拟机主机名）

执行命令
docker stack deploy -c docker-compose.yml pulsar
开始部署（一定要创建好swarm环境，不然会报错）

检查是否成功
docker stack ps pulsar
[root@node-01 zookeeper]# docker stack ps pulsar
ID             NAME                   IMAGE                        NODE      DESIRED STATE   CURRENT STATE               ERROR     PORTS              
b1i4sa8xg9ai   pulsar_zookeeper01.1   apachepulsar/pulsar:latest   node-01   Running         Running 2 hours ago                   
oplsd40z6y1i   pulsar_zookeeper02.1   apachepulsar/pulsar:latest   node-02   Running         Running 2 hours ago                   
03683o820ll2   pulsar_zookeeper03.1   apachepulsar/pulsar:latest   node-03   Running         Running 2 hours ago 
如上所示即成功，并且在其他节点docker ps也有输出
[root@node-02 ~]# docker ps
c369cde09f6f   apachepulsar/pulsar:latest     "bash -c 'bin/apply-…"   2 hours ago         Up 2 hours (healthy)                                               pulsar_zookeeper02.1.oplsd40z6y1izsqm276vptgt9

```

```
version: '3.9'
services:
  # Start zookeeper
  zookeeper01:
    image: apachepulsar/pulsar:latest
    user: root
    hostname: zookeeper01
    # container_name: zookeeper01
    # restart: on-failure
    volumes:
      - zk_01_data:/pulsar/data/zookeeper
    #  - /data/pulsar/data:/pulsar/data/zookeeper
    # networks:
    #  - pulsar
    environment:
      - metadataStoreUrl=zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - PULSAR_MEM=-Xms256m -Xmx256m -XX:MaxDirectMemorySize=256m
      - tickTime=10000
    command: >
      bash -c "bin/apply-config-from-env.py conf/zookeeper.conf
               bin/generate-zookeeper-config.sh conf/zookeeper.conf
               echo 1 > data/zookeeper/myid
               echo 'client.sasl=false' >> conf/zookeeper.conf
               echo 'server.1=zookeeper01:2888:3888' >> conf/zookeeper.conf
               echo 'server.2=zookeeper02:2888:3888' >> conf/zookeeper.conf
               echo 'server.3=zookeeper03:2888:3888' >> conf/zookeeper.conf
               exec bin/pulsar zookeeper"
    healthcheck:
      test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"]
      interval: 10s
      timeout: 5s
      retries: 30
    deploy:
      placement:
        constraints: [ node.hostname == ${node1} ]

  zookeeper02:
    image: apachepulsar/pulsar:latest
    user: root
    hostname: zookeeper02
    # container_name: zookeeper02
    # restart: on-failure
    volumes:
      - zk_02_data:/pulsar/data/zookeeper
    #networks:
    #  - pulsar
    environment:
      - metadataStoreUrl=zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - PULSAR_MEM=-Xms256m -Xmx256m -XX:MaxDirectMemorySize=256m
      - tickTime=10000
    command: >
      bash -c "bin/apply-config-from-env.py conf/zookeeper.conf
               bin/generate-zookeeper-config.sh conf/zookeeper.conf
               echo 2 > data/zookeeper/myid
               echo 'client.sasl=false' >> conf/zookeeper.conf
               echo 'server.1=zookeeper01:2888:3888' >> conf/zookeeper.conf
               echo 'server.2=zookeeper02:2888:3888' >> conf/zookeeper.conf
               echo 'server.3=zookeeper03:2888:3888' >> conf/zookeeper.conf
               exec bin/pulsar zookeeper"
    healthcheck:
      test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"]
      interval: 10s
      timeout: 5s
      retries: 30
    deploy:
      placement:
        constraints: [ node.hostname == ${node2} ]

  zookeeper03:
    image: apachepulsar/pulsar:latest
    user: root
    hostname: zookeeper03
    # container_name: zookeeper03
    # restart: on-failure
    volumes:
      - zk_03_data:/pulsar/data/zookeeper
    #networks:
    #  - pulsar
    environment:
      - metadataStoreUrl=zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - PULSAR_MEM=-Xms256m -Xmx256m -XX:MaxDirectMemorySize=256m
      - tickTime=10000
    command: >
      bash -c "bin/apply-config-from-env.py conf/zookeeper.conf
               bin/generate-zookeeper-config.sh conf/zookeeper.conf
               echo 3 > data/zookeeper/myid
               echo 'client.sasl=false' >> conf/zookeeper.conf
               echo 'server.1=zookeeper01:2888:3888' >> conf/zookeeper.conf
               echo 'server.2=zookeeper02:2888:3888' >> conf/zookeeper.conf
               echo 'server.3=zookeeper03:2888:3888' >> conf/zookeeper.conf
               exec bin/pulsar zookeeper"
    healthcheck:
      test: ["CMD", "bin/pulsar-zookeeper-ruok.sh"]
      interval: 10s
      timeout: 5s
      retries: 30
    deploy:
      placement:
        constraints: [ node.hostname == ${node3} ]

  # Init cluster metadata
  pulsar-init:
    hostname: pulsar-init
    image: apachepulsar/pulsar:latest
    #networks:
    #  - pulsar
    # restart: on-failure
    command: >
      bin/pulsar initialize-cluster-metadata \
               --cluster pulsar-cluster \
               --zookeeper zookeeper01:2181,zookeeper02:2181,zookeeper03:2181 \
               --configuration-store zookeeper01:2181,zookeeper02:2181,zookeeper03:2181 \
               --web-service-url broker01:8080,broker02:8080,broker03:8080 \
               --broker-service-url pulsar://broker01:6650,broker02:6650,broker03:6650
    depends_on:
      - zookeeper01
        #  condition: service_healthy
      - zookeeper02
        #  condition: service_healthy
      - zookeeper03
        #  condition: service_healthy
    deploy:
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 10s
volumes:
  zk_01_data:
    driver: local
  zk_02_data:
    driver: local
  zk_03_data:
    driver: local

```



## 部署Bookkeeper(存储)

Pulsar Bookkeeper是Pulsar的持久化存储组件，用于可靠地存储消息和元数据。它是一个分布式、高性能的日志存储系统，专为处理大规模数据流而设计。

Pulsar Bookkeeper是Pulsar的核心组件之一，用于可靠地存储消息和元数据。它具有分布式存储、日志存储、容错和故障恢复、顺序写入和读取、数据分段和归档等特性，使得Pulsar能够处理大规模数据流，并保证数据的可靠性和持久性。

进入Bookkeeper目录

![image-20230725170016624](https://ye5201314-1312898079.cos.ap-guangzhou.myqcloud.com/image-20230725170016624.png)

编辑docker-compose.yml

```
将constraints: [ node.hostname == node1 ]/constraints: [ node.hostname == node2 ]/constraints: [ node.hostname == node3 ]改为相对应的主机名（所部署的虚拟机主机名）

执行命令
docker stack deploy -c docker-compose.yml pulsar
开始部署（一定要创建好swarm环境，不然会报错）

检查是否成功
docker stack ps pulsar
[root@node-01 zookeeper]# docker stack ps pulsar
ID             NAME                   IMAGE                        NODE      DESIRED STATE   CURRENT STATE               ERROR     PORTS              
11pjjir2h5oz   pulsar_bookie01.1      apachepulsar/pulsar:latest   node-01   Running         Running about an hour ago             
8b286cpdzren   pulsar_bookie02.1      apachepulsar/pulsar:latest   node-02   Running         Running about an hour ago             
wso2xbqzuwv3   pulsar_bookie03.1      apachepulsar/pulsar:latest   node-03   Running         Running about an hour ago  
如上所示即成功，并且在其他节点docker ps也有输出
[root@node-02 ~]# docker ps
3645e070ea8c   apachepulsar/pulsar:latest     "bash -c 'bin/apply-…"   About an hour ago   Up About an hour                                                   pulsar_bookie02.1.8b286cpdzrenrducbfd847ujj
```

```
version: '3'
services:
  # Start bookie
  bookie01:
    image: apachepulsar/pulsar:latest
    user: root
    # container_name: bookie01
    # restart: on-failure
    hostname: bookie01
    environment:
      - clusterName=pulsar-cluster
      - zkServers=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - metadataServiceUrl=metadata-store:zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
#      # otherwise every time we run docker compose uo or down we fail to start due to Cookie
#      # See: https://github.com/apache/bookkeeper/blob/405e72acf42bb1104296447ea8840d805094c787/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Cookie.java#L57-68
      - advertisedAddress=bookie01
      - extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
      - BOOKIE_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
    depends_on:
      - zookeeper01
        #condition: service_healthy
      - zookeeper02
        #condition: service_healthy
      - zookeeper03
        #condition: service_healthy
      #- pulsar-init
        #condition: service_completed_successfully
    # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks.
    volumes:
      - bookie_1_data:/pulsar/data/bookkeeper
    command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && exec bin/pulsar bookie"
    deploy:
      placement:
        constraints: [ node.hostname == node1 ]

  bookie02:
    image: apachepulsar/pulsar:latest
    user: root
    # container_name: bookie02
    # restart: on-failure
    hostname: bookie02
    environment:
      - clusterName=pulsar-cluster
      - zkServers=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - metadataServiceUrl=metadata-store:zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      # otherwise every time we run docker compose uo or down we fail to start due to Cookie
      # See: https://github.com/apache/bookkeeper/blob/405e72acf42bb1104296447ea8840d805094c787/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Cookie.java#L57-68
      - advertisedAddress=bookie02
      - extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
      - BOOKIE_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
    depends_on:
      - zookeeper01
        #condition: service_healthy
      - zookeeper02
        #condition: service_healthy
      - zookeeper03
        #condition: service_healthy
      #- pulsar-init 
        #condition: service_completed_successfully
    # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks.
    volumes:
      - bookie_2_data:/pulsar/data/bookkeeper
    command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && exec bin/pulsar bookie"
    deploy:
      placement:
        constraints: [ node.hostname == node2 ]

  bookie03:
    image: apachepulsar/pulsar:latest
    user: root
    # container_name: bookie03
    # restart: on-failure
    hostname: bookie03
    environment:
      - clusterName=pulsar-cluster
      - zkServers=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - metadataServiceUrl=metadata-store:zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      # otherwise every time we run docker compose uo or down we fail to start due to Cookie
      # See: https://github.com/apache/bookkeeper/blob/405e72acf42bb1104296447ea8840d805094c787/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Cookie.java#L57-68
      - advertisedAddress=bookie03
      - extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
      - BOOKIE_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
    depends_on:
      - zookeeper01
        #condition: service_healthy
      - zookeeper02
        #condition: service_healthy
      - zookeeper03
        #condition: service_healthy
      #- pulsar-init 
        #condition: service_completed_successfully
    # Map the local directory to the container to avoid bookie startup failure due to insufficient container disks.
    volumes:
      - bookie_3_data:/pulsar/data/bookkeeper
    command: bash -c "bin/apply-config-from-env.py conf/bookkeeper.conf && exec bin/pulsar bookie"
    deploy:
      placement:
        constraints: [ node.hostname == node3 ]

volumes:
  bookie_1_data:
    driver: local
  bookie_2_data:
    driver: local
  bookie_3_data:
    driver: local

```



## 部署broker(计算)

Pulsar Broker是Pulsar的核心组件之一，负责接收、存储和分发消息。它是一个高性能、可扩展的消息中间件，用于构建可靠的实时数据流和事件驱动架构。

Pulsar Broker是Pulsar的核心组件之一，负责接收、存储和分发消息。它具有高性能的消息接收和存储能力，支持消息分发和消费，实现了消息的持久化和可靠性。同时，它还支持多租户模式、水平扩展和负载均衡、消息过滤和路由等功能，使得Pulsar能够构建可靠的实时数据流和事件驱动架构

进入的broker目录

![image-20230725170207275](https://ye5201314-1312898079.cos.ap-guangzhou.myqcloud.com/image-20230725170207275.png)

编辑docker-compose.yml

```
将constraints: [ node.hostname == node1 ]/constraints: [ node.hostname == node2 ]/constraints: [ node.hostname == node3 ]改为相对应的主机名（所部署的虚拟机主机名）

执行命令
docker stack deploy -c docker-compose.yml pulsar
开始部署（一定要创建好swarm环境，不然会报错）

检查是否成功
docker stack ps pulsar
[root@node-01 zookeeper]# docker stack ps pulsar
ID             NAME                   IMAGE                        NODE      DESIRED STATE   CURRENT STATE               ERROR     PORTS              
rp62f74nwqjg   pulsar_broker01.1      apachepulsar/pulsar:latest   node-01   Running         Running about an hour ago             
xej0hy7j1dip   pulsar_broker02.1      apachepulsar/pulsar:latest   node-02   Running         Running about an hour ago             
ip2muwdwzafl   pulsar_broker03.1      apachepulsar/pulsar:latest   node-03   Running         Running about an hour ago   
如上所示即成功，并且在其他节点docker ps也有输出
[root@node-02 ~]# docker ps
0c3fbaee0e42   apachepulsar/pulsar:latest     "bash -c 'bin/apply-…"   About an hour ago   Up About an hour                                                   pulsar_broker02.1.xej0hy7j1dipuq8d34z3ihg1i
```

```
version: '3'
services:
  # Start broker
  broker01:
    image: apachepulsar/pulsar:latest
    user: root
    # container_name: broker01
    hostname: broker01
    # restart: on-failure
    environment:
      - metadataStoreUrl=zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - zookeeperServers=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - clusterName=pulsar-cluster
      #- managedLedgerDefaultEnsembleSize=1
      #- managedLedgerDefaultWriteQuorum=1
      #- managedLedgerDefaultAckQuorum=1
      - advertisedAddress=broker01
      #- advertisedListeners=external:pulsar://127.0.0.1:6650
      - PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
      - bindAddress=broker01
    depends_on:
      - zookeeper01
        #condition: service_healthy
      - zookeeper02
        #condition: service_healthy
      - zookeeper03
        #condition: service_healthy
      - bookie01
        #condition: service_started
      - bookie02
        #condition: service_started
      - bookie03
        #condition: service_started
    command: bash -c "bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker"
    deploy:
      placement:
        constraints: [ node.hostname == node1 ]

  broker02:
    image: apachepulsar/pulsar:latest
    user: root
    hostname: broker02
    environment:
      - metadataStoreUrl=zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - zookeeperServers=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - clusterName=pulsar-cluster
      - advertisedAddress=broker02
      - PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
      - bindAddress=broker02
    depends_on:
      - zookeeper01
      - zookeeper02
      - zookeeper03
      - bookie01
      - bookie02
      - bookie03
    command: bash -c "bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker"
    deploy:
      placement:
        constraints: [ node.hostname == node2 ]

  broker03:
    image: apachepulsar/pulsar:latest
    user: root
    hostname: broker03
    environment:
      - metadataStoreUrl=zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - zookeeperServers=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
      - clusterName=pulsar-cluster
      - advertisedAddress=broker03
      - PULSAR_MEM=-Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m
      - bindAddress=broker03
    depends_on:
      - zookeeper01
      - zookeeper02
      - zookeeper03
      - bookie01
      - bookie02
      - bookie03
    command: bash -c "bin/apply-config-from-env.py conf/broker.conf && exec bin/pulsar broker"
    deploy:
      placement:
        constraints: [ node.hostname == node3 ]

```



## proxy（管理）

Proxy是Pulsar的一个组件，用于处理客户端与Pulsar集群之间的通信。它充当了客户端和Pulsar集群之间的网关，负责接收客户端的请求并将其转发到相应的Pulsar Broker节点上。

Proxy的主要作用有以下几点：

1. 路由请求：Proxy接收客户端的请求，并根据请求中的topic信息将请求路由到相应的Pulsar Broker节点。通过Proxy，客户端可以直接与Proxy进行交互，而无需直接与Broker节点通信，简化了客户端的连接管理和负载均衡。
2. 负载均衡：Proxy可以根据当前集群的负载情况，动态地将请求分发到可用的Broker节点上，实现负载均衡。它可以根据Broker节点的负载情况、网络延迟等指标进行智能路由，确保请求能够均匀地分配到各个Broker节点上，提高系统的整体性能和可扩展性。
3. 安全认证：Proxy可以作为安全层，负责处理客户端的身份验证和授权。它可以验证客户端的身份，并根据配置的权限规则来限制客户端的访问权限，保护Pulsar集群的安全。
4. 减少网络开销：Proxy可以通过缓存和压缩等技术，减少客户端与Pulsar集群之间的网络开销。它可以缓存热点数据，提高数据的读取性能，并使用压缩算法减少数据传输的带宽消耗。

总之，Proxy是Pulsar容器集群中的一个重要组件，它负责路由请求、负载均衡、安全认证和减少网络开销等功能。通过使用Proxy，可以简化客户端的连接管理和负载均衡，提高系统的性能、可扩展性和安全性。

```
version: '3.9'
services:
  proxy:
    image: apachepulsar/pulsar:latest
    user: root
    hostname: pulsar-proxy
    environment:
      #- brokerServiceURL=pulsar://broker01:6650,pulsar://broker02:6650,pulsar://broker03:6650
      #- brokerServiceURL=pulsar://broker01:6650
      #- brokerWebServiceURL=broker01:8080,broker02:8080,broker03:8080
      #- brokerWebServiceURL=http://broker01:8080,broker02:8080,broker03:8080
      #- functionWorkerWebServiceURL=broker01:8080,broker02:8080,broker03:8080
      #- advertisedAddress=0.0.0.0
      - metadataStoreUrl=zookeeper01:2181,zookeeper02:2181,zookeeper03:2181
    depends_on:
      - zookeeper01
      - zookeeper02
      - zookeeper03
      - bookie01
      - bookie02
      - bookie03
      - broker01
      - broker02
      - broker03
    ports:
      - "6650:6650"
      - "8080:8080"
    command: >
      bash -c "bin/apply-config-from-env.py conf/proxy.conf
               bin/pulsar proxy"
               #--metadata-store zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181 \
               #--configuration-metadata-store zk:zookeeper01:2181,zookeeper02:2181,zookeeper03:2181"
```

## 验证消息队列

#### 查看Bookkeeper集群

可执行命令验证是否启动集群：

```
docker exec -i --user=root bookkeeper \
bash -c "bin/bookkeeper shell bookiesanity"
```

如果不出意外，会看到成功的提示：

```
INFO  org.apache.bookkeeper.tools.cli.commands.bookie.SanityTestCommand - Bookie sanity test succeeded
```

#### 查看Broker节点

完成启动后，即可通过 `pulsar-admin` 命令查看集群的节点情况：

```
docker exec -i --user=root broker \
bash -c "bin/pulsar-admin --admin-url http://broker01:8080 brokers list pulsar-cluster"
```

如不出意外，可看到终端显示的 broker 节点：

```
"broker01:8080"
"broker02:8080"
"broker03:8080"
```

#### 查看Cluster

**获取 Pulsar Cluster 的部署信息：**

```
docker exec -i --user=root broker \
bash -c "bin/pulsar-admin --admin-url http://broker01:8080 clusters get pulsar-cluster"
```

等待返回结果：

```
{
  "serviceUrl" : "broker01:8080,broker02:8080,broker03:8080",
  "brokerServiceUrl" : "pulsar://broker01:6650,broker02:6650,broker03:6650",
  "brokerClientTlsEnabled" : false,
  "tlsAllowInsecureConnection" : false,
  "brokerClientTlsEnabledWithKeyStore" : false,
  "brokerClientTlsTrustStoreType" : "JKS",
  "brokerClientTlsKeyStoreType" : "JKS"
}
```

**查看 Cluster 列表：**

```
docker exec -i --user=root broker \
bash -c "bin/pulsar-admin --admin-url http://broker01:8080 clusters list"
```

等待返回结果，可看到刚刚创建的集群：

```
pulsar-cluster
```

#### Hello Pulsar

到此，Pulsar 集群已搭建完成，可以进行测试了，使用 `pulsar-client` 命令生产并消费消息。

首先开启一个终端，执行消费监听，这样就可以实时查看到生产的消息，例如计划消费 100 条消息，可在 node-01 上面执行如下命令：

```
docker exec -i --user=root  broker \
bash -c "bin/pulsar-client --url pulsar://broker01:6650 consume persistent://public/default/test   -s 'first-subscription' -n 100"
```

再开一个终端，执行生产消息，例如在 node-02 上面执行如下命令：

```
docker exec -i --user=root broker \
bash -c "bin/pulsar-client --url pulsar://broker01:6650 produce \
  persistent://public/default/test \
  -n 10 \
  -m 'Hello Pulsar'"
```

此时可看到在生产端看到如下输出信息，表示已生产了一条消息：

```
INFO  org.apache.pulsar.client.cli.PulsarClientTool - 1 messages successfully produced
```

回到消费监听端，可看到已经消费了一条消息：

```
----- got message -----
key:[null], properties:[], content:Hello Pulsar
```

![image-20230719111923482](https://ye5201314-1312898079.cos.ap-guangzhou.myqcloud.com/image-20230719111923482.png)

至此，已完成 Pulsar 集群的搭建及测试。

# redis部署

Redis作为缓存可以提高系统的读取性能，减少数据库负载，提升系统的并发能力，支持数据的持久化存储，并提供丰富的数据结构和操作命令。通过合理地使用Redis缓存，可以提高系统的性能和可靠性，提升用户体验

Redis作为缓存的作用主要体现在以下几个方面：

1. 提高读取性能：将热点数据存储在Redis缓存中，可以减轻数据库的读取压力。由于Redis是基于内存的，读取数据的速度非常快，可以大大提高系统的读取性能。当应用程序需要读取数据时，首先会尝试从Redis缓存中获取，如果存在则直接返回，避免了对数据库的访问。
2. 减少数据库负载：通过将部分数据存储在Redis缓存中，可以减少数据库的读写操作，从而减轻数据库的负载。对于一些频繁读取的数据，可以将其缓存到Redis中，避免了每次读取都要查询数据库的开销。这样可以提高数据库的响应速度，并且减少数据库的资源消耗。
3. 提升系统的并发能力：由于Redis是单线程的，可以保证操作的原子性，避免了多线程并发访问时的数据冲突问题。在高并发的场景下，使用Redis作为缓存可以有效地提升系统的并发能力，保证数据的一致性和正确性。
4. 支持数据的持久化：Redis支持数据的持久化存储，可以将缓存中的数据定期或实时地保存到磁盘上，以防止系统故障或重启时数据的丢失。这样即使系统发生异常情况，也可以通过恢复缓存数据来保证系统的正常运行。
5. 提供丰富的数据结构和操作命令：Redis支持多种数据结构和操作命令，可以方便地处理各种类型的数据。例如，可以使用Redis的列表结构来实现消息队列，使用哈希表结构来存储对象属性等。这样可以更灵活地使用Redis来满足不同的业务需求。

进入redis目录

编辑docker-compose.yml

执行命令
docker stack deploy -c docker-compose.yml  redis

开始部署（一定要创建好swarm环境，不然会报错）

```
version: '3'
services:
  redis-node-0:
    image: docker.io/bitnami/redis-cluster:7.0
    volumes:
      - redis-cluster_data-0:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=password'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    deploy:
      placement:
        constraints: [ node.hostname == node1 ]

  redis-node-1:
    image: docker.io/bitnami/redis-cluster:7.0
    volumes:
      - redis-cluster_data-1:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=password'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    deploy:
      placement:
        constraints: [ node.hostname == node1]

  redis-node-2:
    image: docker.io/bitnami/redis-cluster:7.0
    volumes:
      - redis-cluster_data-2:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=password'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    deploy:
      placement:
        constraints: [ node.hostname == node2 ]

  redis-node-3:
    image: docker.io/bitnami/redis-cluster:7.0
    volumes:
      - redis-cluster_data-3:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=password'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    deploy:
      placement:
        constraints: [ node.hostname == node2 ]

  redis-node-4:
    image: docker.io/bitnami/redis-cluster:7.0
    volumes:
      - redis-cluster_data-4:/bitnami/redis/data
    environment:
      - 'REDIS_PASSWORD=password'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
    deploy:
      placement:
        constraints: [ node.hostname == node3 ]

  redis-node-5:
    image: docker.io/bitnami/redis-cluster:7.0
    volumes:
      - redis-cluster_data-5:/bitnami/redis/data
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
    environment:
      - 'REDIS_PASSWORD=password'
      - 'REDISCLI_AUTH=password'
      - 'REDIS_CLUSTER_REPLICAS=1'
      - 'REDIS_NODES=redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5'
      - 'REDIS_CLUSTER_CREATOR=yes'
    deploy:
      placement:
        constraints: [ node.hostname == node3 ]

  redis-cluster-proxy:
    image: kg_redis_cluster_proxy:202307221333
    depends_on:
      - redis-node-0
      - redis-node-1
      - redis-node-2
      - redis-node-3
      - redis-node-4
      - redis-node-5
    ports:
      - 6379:6379
    deploy:
      restart_policy:
        condition: on-failure
    command: >
      src/redis-cluster-proxy -a password --port 6379 redis-node-0:6379,redis-node-1:6379,redis-node-2:6379,redis-node-3:6379,redis-node-4:6379,redis-node-5:6379

volumes:
  redis-cluster_data-0:
    driver: local
  redis-cluster_data-1:
    driver: local
  redis-cluster_data-2:
    driver: local
  redis-cluster_data-3:
    driver: local
  redis-cluster_data-4:
    driver: local
  redis-cluster_data-5:
    driver: local

```

**成功添加后登录redis查看**

```
redis-cli -c -a r@WPbVETNb4o5 -h 192.168.30.12 -p 6376
```

**查看集群信息**

```
cluster info
```

**显示cluster_state:ok就是创建成功：如下图所示**

![image-20230712161126034](https://ye5201314-1312898079.cos.ap-guangzhou.myqcloud.com/image-20230712161126034.png)