Comprehensive Guide to FastDFS Cluster Deployment

Tue, 04 Jul 2023 00:00:00 +0000

Reference

https://www.cnblogs.com/lincappu/p/14926757.html

FastDFS

The author of FastDFS, Yu Qing, described it on GitHub as follows: “FastDFS is an open source high performance distributed file system. It’s major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and video sharing site.” This means FastDFS is an open-source high-performance distributed file system. Its core functionalities include file storage, file synchronization, and file access (uploading and downloading), addressing challenges of large-scale storage and load balancing. It is designed to meet the needs of file-service-oriented websites like photo or video sharing platforms.

FastDFS has two roles: Tracker and Storage. The Tracker is responsible for scheduling file access requests and load balancing. Storage manages file operations, including storage, synchronization, and providing file access interfaces. It also handles metadata, represented as key-value attribute pairs associated with files. Both Tracker and Storage nodes can be composed of one or multiple servers. Servers can be added or removed without disrupting services, though at least one server in each cluster must remain operational. Notably, all servers in a Tracker cluster operate in a peer-to-peer (P2P) manner, allowing dynamic scaling based on server workload.

Additionally, the official documentation elaborates on the storage architecture. To support massive capacity, the Storage nodes adopt a volume (or group) organizing approach. The storage system consists of one or more independent volumes, where the total system capacity equals the sum of all volumes. Each volume can be hosted by one or multiple Storage servers. All servers within the same volume store identical files, serving purposes of redundant backup and load balancing. Adding servers to a volume…

When adding a new server, the system automatically performs file synchronization. Once synchronization is complete, the system automatically switches the new server online to provide services. When storage space is insufficient or near depletion, volumes can be dynamically expanded. Simply add one or more servers and configure them as new volumes to increase the storage system’s capacity. We will not delve too deeply into the concepts of volumes or groups here, as there will be detailed explanations in the subsequent installation and deployment.

In FastDFS, the file identifier consists of two parts: the volume name and the file name.

Environment Specifications

Operating System: CentOS Linux release 7.2.1511
System Disk: 274GB
Mounted Disks: 3.7TB * 12
CPU: 32 cores (Intel® Xeon®)
Memory: 8GB

Architecture Design

Workflow:

The client sends a request to the Tracker.
The Tracker retrieves metadata from Storage nodes and returns it to the client.
The client uses the metadata to directly request files from Storage nodes.

Key Design Principles

The core system contains two roles: Tracker Server and Storage Server.
All Tracker servers are peer-to-peer (P2P) with no Master-Slave relationships.
Storage servers are organized into groups; files are fully replicated within the same group.
Storage servers across different groups do not communicate. Synchronization occurs only within the same group.
Storage servers proactively report status to Trackers. Each Tracker maintains complete Storage server status records.
When the Trunk feature is enabled, Trackers coordinate with Storages to elect a Trunk-Server.

Cluster Deployment

Table 1 Software List and Versions

Name	Description	Link
CentOS	7.x (Installation OS)
libfastcommon	Utility function package for FastDFS	libfastcommon V1.0.39
FastDFS	FastDFS Main Program	FastDFS V5.11
fastdfs-nginx-module	FastDFS-Nginx integration module (resolves intra-group sync delay)	fastdfs-nginx-module V1.22
nginx	nginx 1.12.2 (Latest version via YUM for CentOS 7)	nginx 1.17.4

Table 2 Server IPs, Service Allocation, and Port Planning

Name	IP Address	Application Service	Port
Machine A
	10.58.10.136	tracker	22122
	10.58.10.136	storage-group1	23000
	10.58.10.136	storage-group2	33000
	10.58.10.136	libfastcommon	-
	10.58.10.136	nginx	8888
	10.58.10.136	fastdfs-nginx-module	-
Machine B
	10.58.10.137	tracker	22122

| | 10.58.10.137 | storage-group1 | 23000 | | | 10.58.10.137 | storage-group3 | 43000 | | | 10.58.10.137 | libfastcommon | - | | | 10.58.10.137 | nginx | 8888 | | | 10.58.10.137 | fastdfs-nginx-module | - | |Machine C| | | | | | 10.58.10.138 | tracker | 22122 | | | 10.58.10.138 | storage-group2 | 33000 | | | 10.58.10.138 | storage-group23 | 43000 | | | 10.58.10.138 | libfastcommon | - | | | 10.58.10.138 | nginx | 8888 | | | 10.58.10.138 | fastdfs-nginx-module | - |

Prerequisites before installation:

Ensure to grant read and write permissions to the storage directories (logs, data, PID files, etc.) that will be used.
All configuration files in the following sections contain comments marked with “#” for explanations. Be sure to remove these comment lines that start with “#” when implementing the configurations.

Initialize Environment

1
2

# Install build environment
yum groups install Development Tools perl redhat-rpm-config.noarch gd-devel perl-devel perl-ExtUtils-Embed pcre-devel openssl openssl-devel gcc-c++ autoconf automake zlib-devel libxml2 libxml2-dev libxslt-devel GeoIP GeoIP-devel GeoIP-data gperftools-y

Install libfastcommon

Execute the following operations on machines A, B, and C respectively:

tar -zxvf libfastcommon-1.0.39.tar.gz
cd libfastcommon-1.0.39/
./make.sh
./make.sh install

libfastcommon is installed to /usr/lib64/libfastcommon.so. Note the difference between new and old versions:

New versions will automatically create a symlink for libfastcommon.so to the /usr/local/lib directory.
Old versions require manual symlink creation:

1
2

ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so

If libfdfsclient.so exists, also add it to /usr/local/lib:

1
2

ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so

!!! Note It’s recommended to manually verify successful symlink creation using:

`1`	`ls \| grep libfastcommon`

Check in both `/usr/lib/` and `/usr/local/lib` directories.

Install Tracker

Perform the following operations on machines A, B, and C respectively

mkdir -p /data/fastdfs/tracker
tar -zxvf fastdfs-5.11.tar.gz
cd fastdfs-5.11/
./make.sh
./make.sh install
# Prepare configuration files
cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf # For tracker node

Modify tracker configuration file:

vim /etc/fdfs/tracker.conf
# Required modifications:
max_connections=1024 # default 256, maximum number of connections
port=22122 # tracker server port (default 22122, usually unchanged)
base_path=/data/fastdfs/tracker # root directory for storing logs and data

Add tracker.service to enable service management with systemctl (start/restart/stop) and auto-start on boot:

# Edit service file
vim /usr/lib/systemd/system/fastdfs-tracker.service
[Unit]
Description=The FastDFS File server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
ExecStart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
ExecStop=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop
ExecRestart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart

[Install]
WantedBy=multi-user.target

After saving the /usr/lib/systemd/system/fastdfs-tracker.service file and exiting vim, execute the following commands to start the FastDFS Tracker service:

1
2
3

$ systemctl daemon-reload  
$ systemctl enable fastdfs-tracker.service  
$ systemctl start fastdfs-tracker.service

After the tracker service starts, use the following command to verify if the port is properly opened:

`1`	`netstat -tulnp \| grep 22122 # Check if the service is running and the port is open`

Install Storage

Skip the extraction step if already performed during tracker installation.

$ tar -zxvf fastdfs-5.11.tar.gz  
$ cd fastdfs-5.11/  
$ ./make.sh  
$ ./make.sh install

Machine A (group1/group2)

Copy storage configuration files under fastdfs-5.11 directory (make two copies)

sudo mkdir -p /data/fastdfs/storage/group1
sudo mkdir -p /data/fastdfs/storage/group2
sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage-group1.conf #storage node group1
sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage-group2.conf #storage node group2
sudo cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf #client config for testing

According to architecture design, modify the three files sequentially:

Modify group1 configuration file: /etc/fdfs/storage-group1.conf

$ sudo vim /etc/fdfs/storage-group2.conf
# The following content needs to be modified:
group_name=group2
port=33000 # Storage service port (default 23000, modified to 33000)
base_path=/data/fastdfs/storage/group2 # Root directory for data and log file storage
store_path_count=6
store_path0=/data07/fastdfs # First storage directory for group2
store_path1=/data08/fastdfs # Second storage directory for group2
store_path2=/data09/fastdfs # Third storage directory for group2
store_path3=/data10/fastdfs # Fourth storage directory for group2
store_path4=/data11/fastdfs # Fifth storage directory for group2
store_path5=/data12/fastdfs # Sixth storage directory for group2
tracker_server=10.58.10.136:22122 # Tracker server IP and port
tracker_server=10.58.10.137:22122 # Tracker server IP and port
tracker_server=10.58.10.138:22122 # Tracker server IP and port
http.server_port=8888 # HTTP access port for files (default 8888, modify as needed to match Nginx configuration)

Modify client configuration file: /etc/fdfs/client.conf

$ sudo vim /etc/fdfs/client.conf
# The following content needs to be modified:
base_path=/data/fastdfs/client
tracker_server=10.58.10.136:22122 # Tracker server IP and port
tracker_server=10.58.10.137:22122 # Tracker server IP and port
tracker_server=10.58.10.138:22122 # Tracker server IP and port

After setting up the Storage service, we start the two services for Storage:

1
2
3

$ systemctl daemon-reload
$ systemctl enable fastdfs-storage-group1.service
$ systemctl start fastdfs-storage-group1.service

Possible issues during startup:

The service might fail to start due to permission issues or configuration errors.
Check service status using:
systemctl status fastdfs-storage-group1.service
Analyze logs (located at /data/fastdfs/storage/group1/logs/) to troubleshoot.
Refer to the “Troubleshooting” section for common pitfalls.

Verify service activation:

`1`	`netstat -tulnp # Check if services are running and ports are open (23000, 33000)`

Check FastDFS cluster status after successful startup:

1
2

# View cluster status
$ fdfs_monitor /etc/fdfs/storage-group1.conf list

??? note “Info”
[Cluster status details will be shown here]

The console printed the following information, indicating success: [2018-11-06 00:00:00] DEBUG - base_path=/data/fastdfs/storage/group1, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=3, server_index=0 tracker server is 10.58.10.136:22122,10.58.10.137:22122,10.58.10.138:22122 group count: 2 Group 1: …

Test file upload via client

`1`	`$ fdfs_upload_file /etc/fdfs/client.conf test.txt`

Machine B (group1/group3)

The configuration process is similar to Machine A, with the following modifications:

Create directories and copy configuration files

$ sudo mkdir -p /data/fastdfs/storage/group1  
$ sudo mkdir -p /data/fastdfs/storage/group3  
$ sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage-group1.conf  # Storage node configuration for group1  
$ sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage-group3.conf  # Storage node configuration for group3  
$ sudo cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf          # Client configuration for testing

Modify the configuration file for group3 (/etc/fdfs/storage-group3.conf). The configuration for group1 remains consistent with Machine A.

$ sudo vim /etc/fdfs/storage-group3.conf
# Required modifications are as follows:
group_name=group3
port=43000 # Storage service port (default: 23000, modified to 43000)
base_path=/data/fastdfs/storage/group3 # Root directory for data and log files
store_path_count=6
store_path0=/data07/fastdfs # First storage directory for group3
store_path1=/data08/fastdfs # Second storage directory for group3
store_path2=/data09/fastdfs # Third storage directory for group3
store_path3=/data10/fastdfs # Fourth storage directory for group3
store_path4=/data11/fastdfs # Fifth storage directory for group3
store_path5=/data12/fastdfs # Sixth storage directory for group3
tracker_server=10.58.10.136:22122 # Tracker server IP and port
tracker_server=10.58.10.137:22122 # Tracker server IP and port
tracker_server=10.58.10.138:22122 # Tracker server IP and port
http.server_port=8888 # HTTP file access port (default: 8888, keep consistent with Nginx)

The client configuration remains consistent with Machine A, no duplication needed here.
Create a service for starting group3. The configuration for fastdfs-storage-group1.service is identical to Machine A - simply copy it.
Execute the startup script until both fastdfs-storage-group1.service and fastdfs-storage-group3.service are running.

Configuration for Machine C (Group2/Group3)

The configuration process is similar to Machine A, with the following modifications:

Create directories and copy configuration files:

$ sudo mkdir -p /data/fastdfs/storage/group2
$ sudo mkdir -p /data/fastdfs/storage/group3
$ sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage-group2.conf  # Storage node for group2
$ sudo cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage-group3.conf  # Storage node for group3
$ sudo cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf  # Client configuration file (for testing)

Modify the configuration file for Group2 (/etc/fdfs/storage-group2.conf). The configuration content for Group3 should remain consistent with Machine B’s settings.

Configuration file for FastDFS Storage group2

$ sudo vim /etc/fdfs/storage-group2.conf

Modified configurations:

group_name=group2 port=33000 # Storage service port (default 23000, modified to 33000) base_path=/data/fastdfs/storage/group2 # Root directory for data and log storage store_path_count=6 store_path0=/data01/fastdfs # First storage directory for group2 store_path1=/data02/fastdfs # Second storage directory for group2 store_path2=/data03/fastdfs # Third storage directory for group2 store_path3=/data04/fastdfs # Fourth storage directory for group2 store_path4=/data05/fastdfs # Fifth storage directory for group2 store_path5=/data06/fastdfs # Sixth storage directory for group2 tracker_server=10.58.10.136:22122 # Tracker server IP and port tracker_server=10.58.10.137:22122 # Tracker server IP and port tracker_server=10.58.10.138:22122 # Tracker server IP and port http.server_port=8888 # HTTP file access port (default 8888, should match nginx configuration)

The client configuration file is consistent with Machine A, no repetition needed here.
Create startup services for group2 and group3 (existing templates can be copied directly).
Execute startup scripts until both fastdfs-storage-group2.service and fastdfs-storage-group3.services are running.

Installing Nginx and the FastDFS Nginx Module

Step 1: In the FastDFS directory, copy the http.conf and mime.types files to the /etc/fdfs directory to enable Nginx access to the Storage service.

1
2
3

# Execute on all three machines
$ cp ./conf/http.conf /etc/fdfs/ # For Nginx access
$ cp ./conf/mime.types /etc/fdfs/ # For Nginx access

Step 2: Install the FastDFS Nginx module:

1
2
3

# Execute on all three machines
$ tar -zxvf V1.20.tar.gz
$ cp fastdfs-nginx-module-1.20/src/mod_fastdfs.conf /etc/fdfs/mod_fastdfs.conf

Step 3: Modify the configuration file in fastdfs-nginx-module-1.20/src/config. Locate the ngx_module_incs and CORE_INCS entries and modify them as follows:

1
2

ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/"
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"

If not modified, the following error will occur during Nginx compilation:
/usr/include/fastdfs/fdfs_define.h:15:27: fatal error: common_define.h: No such file or directory

Step 4: Then extract and install the Nginx service:

$ tar -zxvf nginx-1.12.2.tar.gz $ cd nginx-1.12.2 $ ./configure –prefix=/usr/share/nginx –sbin-path=/usr/sbin/nginx –modules-path=/usr/lib64/nginx/modules –conf-path=/etc/nginx/nginx.conf –error-log-path=/var/log/nginx/error.log –http-log-path=/var/log/nginx/access.log –http-client-body-temp-path=/var/lib/nginx/tmp/client_body –http-proxy-temp-path=/var/lib/nginx/tmp/proxy –http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi –http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi –http-scgi-temp-path=/var/lib/nginx/tmp/scgi –pid-path=/run/nginx.pid –lock-path=/run/lock/subsys/nginx –user=nginx –group=nginx –with-file-aio –with-ipv6 –with-http_auth_request_module –with-http_ssl_module –with-http_v2_module –with-http_realip_module –with-http_addition_module –with-http_xslt_module=dynamic –with-http_image_filter_module=dynamic –with-http_geoip_module=dynamic –with-http_sub_module –with-http_dav_module –with-http_flv_module –with-http_mp4_module –with-http_gunzip_module –with-http_gzip_static_module –with-http_random_index_module –with-http_secure_link_module –with-http_degradation_module –with-http_slice_module –with-http_stub_status_module –with-http_perl_module=dynamic –with-mail=dynamic –with-mail_ssl_module –with-pcre –with-pcre-jit –with-stream=dynamic –with-stream_ssl_module –with-google_perftools_module –with-debug –with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong –param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic’ –with-ld-opt=’-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E’ –add-module=${YOUR_PATH}/fastdfs-nginx-module-1.20/src $ make $ make install

Note: Replace ${YOUR_PATH} in the above with the parent directory of fastdfs-nginx-module-1.20 to ensure the path is correct.

Modify configuration file /etc/fdfs/mod_fastdfs.conf on Machine A:

connect_timeout=2
network_timeout=30
base_path=/data/fastdfs/ngx_mod
load_fdfs_parameters_from_tracker=true
storage_sync_file_max_delay = 86400
use_storage_id = false
storage_ids_filename = storage_ids.conf
tracker_server=10.58.10.136:22122 # Tracker server IP and port
tracker_server=10.58.10.137:22122 # Tracker server IP and port
tracker_server=10.58.10.138:22122 # Tracker server IP and port
group_name=group1/group2 # Global setting
url_have_group_name = true
log_level=info
log_filename=
response_mode=proxy
if_alias_prefix=
flv_support = true
flv_extension = flv
group_count = 2

[group1]
group_name=group1 # Group-specific
storage_server_port=23000
store_path_count=6
store_path0=/data01/fastdfs
store_path1=/data02/fastdfs
store_path2=/data03/fastdfs
store_path3=/data04/fastdfs
store_path4=/data05/fastdfs
store_path5=/data06/fastdfs

[group2]
group_name=group2
storage_server_port=33000
store_path_count=6
store_path0=/data07/fastdfs
store_path1=/data08/fastdfs
store_path2=/data09/fastdfs
store_path3=/data10/fastdfs
store_path4=/data11/fastdfs
store_path5=/data12/fastdfs

user nginx; worker_processes on; worker_rlimit_nofile 65535;

error_log /var/log/nginx/error.log; pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events { worker_connections 65535; use epoll; accept_mutex off; }

http { log_format main ‘$remote_addr - $remote_user [$time_local] “$request” ’ ‘$status $body_bytes_sent “$http_referer” ’ ‘"$http_user_agent" “$http_x_forwarded_for”’;

access_log  /var/log/nginx/access.log  main;

sendfile            on;
tcp_nopush          on;
tcp_nodelay         on;
keepalive_timeout   65;
types_hash_max_size 2048;
gzip                on;

server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
larger_client_header_buffers 4 32k;
client_max_body_size 300m;

proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Connection '';
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout   90;
proxy_read_timeout   90;
proxy_buffer_size 16k;
proxy_buffers 8 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;

proxy_cache_path /data/fastdfs/cache/nginx/proxy_cache levels=1:2
keys_zone=http-cache:200m max_size=1g inactive=30d;
proxy_temp_path /data/fastdfs/cache/nginx/proxy_cache/temp;

include             /etc/nginx/mime.types;
default_type        application/octet-stream;

include /etc/nginx/conf.d/*.conf;

server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  _;
    root         /usr/share/nginx/html;

    # Load default server block configuration
    #include /etc/nginx/default.d/*.conf;

    location ~ ^/ok(\..*)?$ {
        return 200 "OK";
    }

    location /nginx {
        stub_status on;
    }

    location /healthcheck {
        check_status on;
    }

    location ^~ /group1/ {
        proxy_next_upstream http_502 http_504 error timeout invalid_header;
        proxy_cache http-cache;
        proxy_cache_valid 200 304 12h;
        proxy_cache_key $uri$is_args$args;

        add_header 'Access-Control-Allow-Origin' $http_origin;
        add_header 'Access-Control-Allow-Credentials' 'true';
        add_header "Access-Control-Allow-Methods" "GET, POST, HEAD, PUT, DELETE, OPTIONS, PATCH";
        add_header "Access-Control-Allow-Headers" "Origin, No-Cache, Authorization, X-Requested-With, If-Modified-Since, Pragma, Last-Modified, Cache-Control, Expires, Content-Type";
        if ($request_method = 'OPTIONS') {
            return 200 'OK';
        }

        proxy_pass http://fdfs_group1;
        expires 30d;
    }
    location ^~ /group2/ {
        proxy_next_upstream http_502 http_504 error timeout invalid_header;
        proxy_cache http-cache;
        proxy_cache_valid 200 304 12h;
        proxy_cache_key $uri$is_args$args;

        add_header 'Access-Control-Allow-Origin' $http_origin;
        add_header 'Access-Control-Allow-Credentials' 'true';
        add_header "Access-Control-Allow-Methods" "GET, POST, HEAD, PUT, DELETE, OPTIONS, PATCH";
        add_header "Access-Control-Allow-Headers" "Origin, No-Cache, Authorization, X-Requested-With, If-Modified-Since, Pragma, Last-Modified, Cache-Control, Expires, Content-Type";
        if ($request_method = 'OPTIONS') {
            return 200 'OK';
        }

        proxy_pass http://fdfs_group2;
        expires 30d;
    }
    location ^~ /group3/ {
        proxy_next_upstream http_502 http_504 error timeout invalid_header;
        proxy_cache http-cache;
        proxy_cache_valid 200 304 12h;
        proxy_cache_key $uri$is_args$args;

        add_header 'Access-Control-Allow-Origin' $http_origin;
        add_header 'Access-Control-Allow-Credentials' 'true';
        add_header "Access-Control-Allow-Methods" "GET, POST, HEAD, PUT, DELETE, OPTIONS, PATCH";
        add_header "Access-Control-Allow-Headers" "Origin, No-Cache, Authorization, X-Requested-With, If-Modified-Since, Pragma, Last-Modified, Cache-Control, Expires, Content-Type";
        if ($request_method = 'OPTIONS') {
            return 200 'OK';
        }

        proxy_pass http://fdfs_group3;
        expires 30d;
    }

    location ~/purge(/.*) {
        allow 127.0.0.1;
        allow 192.168.1.0/24;
        allow 10.58.1.0/24;
        deny all;
        proxy_cache_purge http-cache $1$is_args$args;
    }
}

server {
    listen 8888;
    server_name localhost;
    
    location /ok.htm {
        return 200 "OK";
    }

    location ~/group[0-9]/ {
        ngx_fastdfs_module;
    }

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root html;
    }
}

upstream fdfs_group1 {
    server 10.58.10.136:8888 max_fails=0;
    server 10.58.10.137:8888 max_fails=0;

    keepalive 10240;
    check interval=2000 rise=2 fall=3 timeout=1000 type=http default_down=false;
    check_http_send "GET /ok.htm HTTP/1.0\r\nConnection:keep-alive\r\n\r\n";
    check_keepalive_requests 100;
}

upstream fdfs_group2 {
    server 10.58.10.136:8888 max_fails=0;
    server 10.58.10.138:8888 max_fails=0;

    keepalive 10240;
    check interval=2000 rise=2 fall=3 timeout=1000 type=http default_down=false;
    check_http_send "GET /ok.htm HTTP/1.0\r\nConnection:keep-alive\r\n\r\n";
    check_keepalive_requests 100;
}

upstream fdfs_group3 {
    server 10.58.10.137:8888 max_fails=0;
    server 10.58.10.138:8888 max_fails=0;

    keepalive 10240;
    check interval=2000 rise=2 fall=3 timeout=1000 type=http default_down=false;
    check_http_send "GET /ok.htm HTTP/1.0\r\nConnection:keep-alive\r\n\r\n";
    check_keepalive_requests 100;
}

}

Start the nginx service:

sudo nginx -c /etc/nginx/nginx.conf

Access http://localhost/ok.htm to verify if it returns a 200 status code with “OK” content. If startup fails, check the nginx logs located at /var/log/nginx/error.log. This is the default error log path for nginx in CentOS. If the error_log configuration has been modified, check the log file specified in the error_log directive.

The configurations for servers B and C are essentially identical, with completely consistent nginx configuration files. The only required modification is in the /etc/fdfs/mod_fastdfs.conf file: adjust the group_name according to the corresponding storage node configuration. Note that the [groupX] identifiers in square brackets must follow sequential increments starting from 1.

Issue Summary

When encountering issues, first, we troubleshoot service startup failures through console log messages; second, investigate via service logs. In most cases, logs are the most effective troubleshooting tool, bar none.

Logs for tracker or storage services are stored under the logs directory within the path specified by the base_path configuration in their respective service configuration files.
For nginx logs, if a custom log path is configured, check the specified directory. By default on CentOS, logs are typically located under /var/log/nginx/.
Additionally, in this environment, we must also investigate logs for the fastdfs-nginx-module extension. Its logs reside in the directory specified by the base_path configuration in /etc/fdfs/mod_fastdfs.conf.

Port Configuration for Multiple Groups in a Cluster

Each group requires a dedicated storage service. Thus, port numbers for storage groups on the same host must not conflict. For example:
- Group1: Port 23000
- Group2: Port 33000
- Group3: Port 43000
Groups with the same name cannot exist on the same machine because the tracker synchronizes storage services within the same group, which requires identical port numbers. Therefore, groups with the same name across different hosts must use the same storage port configuration.

Storage Path Configuration for Multiple Groups in a Cluster

On the same host, storage paths for different groups must be configured separately and should not overlap.
Across different hosts, storage paths for groups with the same name may differ, but the number of storage nodes and disk capacity should remain consistent.

Configuring Multiple Storage Services on the Same Node

In a 3-node cluster, two groups were deployed on the same machine, with each group maintaining a replica on another node. When starting the services using systemctl start fastdfs-storage-groupx.service (where groupx represents group1, group2, or group3), one group repeatedly failed to start, displaying:

1
2
3

Loaded: loaded (/usr/lib/systemd/system/fastdfs-storage-group1.service; enabled; vendor preset: disabled)  
...  
Active: inactive (dead) ➔ Active: exited

Resolution: A systemctl daemon-reload is required before starting each storage service.
Example: To start both fastdfs-storage-group1.service and fastdfs-storage-group2.service on Machine A:

Start fastdfs-storage-group1.service:

1
2

   systemctl daemon-reload  
   systemctl start fastdfs-storage-group1.service

Start fastdfs-storage-group2.service:

1
2

   systemctl daemon-reload  
   systemctl start fastdfs-storage-group2.service

FastDFS Storage Service Configuration
On all storage machines, when modifying /etc/fdfs/mod_fastdfs.conf, adjust the group_name value according to the respective group of each storage. Pay special attention to two configuration items. Below is an example using group2 and group3:

Global group_name
The group_name in the global configuration must be separated by “/”, and its value should match the group_name in local configurations. For example:

`1`	`group_name=group2/group3`

Steps to Configure and Start Services

Execute the following commands for group1:

1
2
3

   sudo systemctl daemon-reload
   sudo systemctl enable fastdfs-storage-group1.service
   sudo systemctl start fastdfs-storage-group1.service

Check the status of fastdfs-storage-group1.service:

`1`	`systemctl status fastdfs-storage-group1.service`

If the status shows active (running), proceed to start fastdfs-storage-group2.service. If not, check logs to troubleshoot:

1
2
3

   sudo systemctl daemon-reload
   sudo systemctl enable fastdfs-storage-group2.service
   sudo systemctl start fastdfs-storage-group3.service  # Note: Verify group name consistency (group2 vs group3)

Monitor the status of fastdfs-storage-group2.service until it becomes active.

Notes

You may encounter a warning like Unknown lvalue 'ExecRestart' in section 'Service'. A known workaround (yum install systemd-*) might not resolve this issue. Contributions to fix this are welcome.

FastDFS Nginx Module Configuration
When modifying /etc/fdfs/mod_fastdfs.conf on storage machines:

Ensure group_name values are correctly set for each storage group (e.g., group2 and group3).
Follow the global/local group_name format as described above.

<group_entries> </group_entries>

Distributed Storage on Kang's Blog