Reference
https://www.cnblogs.com/lincappu/p/14926757.html
FastDFS
The author of FastDFS, Yu Qing, described it on GitHub as follows: “FastDFS is an open source high performance distributed file system. It’s major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and video sharing site.” This means FastDFS is an open-source high-performance distributed file system. Its core functionalities include file storage, file synchronization, and file access (uploading and downloading), addressing challenges of large-scale storage and load balancing. It is designed to meet the needs of file-service-oriented websites like photo or video sharing platforms.
FastDFS has two roles: Tracker and Storage. The Tracker is responsible for scheduling file access requests and load balancing. Storage manages file operations, including storage, synchronization, and providing file access interfaces. It also handles metadata, represented as key-value attribute pairs associated with files. Both Tracker and Storage nodes can be composed of one or multiple servers. Servers can be added or removed without disrupting services, though at least one server in each cluster must remain operational. Notably, all servers in a Tracker cluster operate in a peer-to-peer (P2P) manner, allowing dynamic scaling based on server workload.
Additionally, the official documentation elaborates on the storage architecture. To support massive capacity, the Storage nodes adopt a volume (or group) organizing approach. The storage system consists of one or more independent volumes, where the total system capacity equals the sum of all volumes. Each volume can be hosted by one or multiple Storage servers. All servers within the same volume store identical files, serving purposes of redundant backup and load balancing. Adding servers to a volume…
When adding a new server, the system automatically performs file synchronization. Once synchronization is complete, the system automatically switches the new server online to provide services. When storage space is insufficient or near depletion, volumes can be dynamically expanded. Simply add one or more servers and configure them as new volumes to increase the storage system’s capacity. We will not delve too deeply into the concepts of volumes or groups here, as there will be detailed explanations in the subsequent installation and deployment.
In FastDFS, the file identifier consists of two parts: the volume name and the file name.
Environment Specifications
- Operating System: CentOS Linux release 7.2.1511
- System Disk: 274GB
- Mounted Disks: 3.7TB * 12
- CPU: 32 cores (Intel® Xeon®)
- Memory: 8GB
Architecture Design
Workflow:
- The client sends a request to the Tracker.
- The Tracker retrieves metadata from Storage nodes and returns it to the client.
- The client uses the metadata to directly request files from Storage nodes.
Key Design Principles
- The core system contains two roles: Tracker Server and Storage Server.
- All Tracker servers are peer-to-peer (P2P) with no Master-Slave relationships.
- Storage servers are organized into groups; files are fully replicated within the same group.
- Storage servers across different groups do not communicate. Synchronization occurs only within the same group.
- Storage servers proactively report status to Trackers. Each Tracker maintains complete Storage server status records.
- When the Trunk feature is enabled, Trackers coordinate with Storages to elect a Trunk-Server.
Cluster Deployment
Table 1 Software List and Versions
| Name | Description | Link |
|---|---|---|
| CentOS | 7.x (Installation OS) | |
| libfastcommon | Utility function package for FastDFS | libfastcommon V1.0.39 |
| FastDFS | FastDFS Main Program | FastDFS V5.11 |
| fastdfs-nginx-module | FastDFS-Nginx integration module (resolves intra-group sync delay) | fastdfs-nginx-module V1.22 |
| nginx | nginx 1.12.2 (Latest version via YUM for CentOS 7) | nginx 1.17.4 |
Table 2 Server IPs, Service Allocation, and Port Planning
| Name | IP Address | Application Service | Port |
|---|---|---|---|
| Machine A | |||
| 10.58.10.136 | tracker | 22122 | |
| 10.58.10.136 | storage-group1 | 23000 | |
| 10.58.10.136 | storage-group2 | 33000 | |
| 10.58.10.136 | libfastcommon | - | |
| 10.58.10.136 | nginx | 8888 | |
| 10.58.10.136 | fastdfs-nginx-module | - | |
| Machine B | |||
| 10.58.10.137 | tracker | 22122 |
| | 10.58.10.137 | storage-group1 | 23000 | | | 10.58.10.137 | storage-group3 | 43000 | | | 10.58.10.137 | libfastcommon | - | | | 10.58.10.137 | nginx | 8888 | | | 10.58.10.137 | fastdfs-nginx-module | - | |Machine C| | | | | | 10.58.10.138 | tracker | 22122 | | | 10.58.10.138 | storage-group2 | 33000 | | | 10.58.10.138 | storage-group23 | 43000 | | | 10.58.10.138 | libfastcommon | - | | | 10.58.10.138 | nginx | 8888 | | | 10.58.10.138 | fastdfs-nginx-module | - |
Prerequisites before installation:
-
Ensure to grant read and write permissions to the storage directories (logs, data, PID files, etc.) that will be used.
-
All configuration files in the following sections contain comments marked with “#” for explanations. Be sure to remove these comment lines that start with “#” when implementing the configurations.
Initialize Environment
|
|
Install libfastcommon
Execute the following operations on machines A, B, and C respectively:
|
|
libfastcommon is installed to /usr/lib64/libfastcommon.so. Note the difference between new and old versions:
- New versions will automatically create a symlink for
libfastcommon.soto the/usr/local/libdirectory. - Old versions require manual symlink creation:
|
|
If libfdfsclient.so exists, also add it to /usr/local/lib:
|
|
!!! Note It’s recommended to manually verify successful symlink creation using:
|
|
Check in both `/usr/lib/` and `/usr/local/lib` directories.
Install Tracker
Perform the following operations on machines A, B, and C respectively
|
|
Modify tracker configuration file:
|
|
Add tracker.service to enable service management with systemctl (start/restart/stop) and auto-start on boot:
|
|
After saving the /usr/lib/systemd/system/fastdfs-tracker.service file and exiting vim, execute the following commands to start the FastDFS Tracker service:
|
|
After the tracker service starts, use the following command to verify if the port is properly opened:
|
|
Install Storage
Skip the extraction step if already performed during tracker installation.
|
|
Machine A (group1/group2)
Copy storage configuration files under fastdfs-5.11 directory (make two copies)
|
|
According to architecture design, modify the three files sequentially:
- Modify group1 configuration file: /etc/fdfs/storage-group1.conf
|
|
- Modify client configuration file: /etc/fdfs/client.conf
|
|
- After setting up the Storage service, we start the two services for Storage:
|
|
Possible issues during startup:
- The service might fail to start due to permission issues or configuration errors.
- Check service status using:
systemctl status fastdfs-storage-group1.service - Analyze logs (located at
/data/fastdfs/storage/group1/logs/) to troubleshoot. - Refer to the “Troubleshooting” section for common pitfalls.
Verify service activation:
|
|
- Check FastDFS cluster status after successful startup:
|
|
??? note “Info”
[Cluster status details will be shown here]
The console printed the following information, indicating success: [2018-11-06 00:00:00] DEBUG - base_path=/data/fastdfs/storage/group1, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=3, server_index=0 tracker server is 10.58.10.136:22122,10.58.10.137:22122,10.58.10.138:22122 group count: 2 Group 1: …
- Test file upload via client
|
|
Machine B (group1/group3)
The configuration process is similar to Machine A, with the following modifications:
- Create directories and copy configuration files
|
|
- Modify the configuration file for group3 (
/etc/fdfs/storage-group3.conf). The configuration for group1 remains consistent with Machine A.
|
|
-
The client configuration remains consistent with Machine A, no duplication needed here.
-
Create a service for starting group3. The configuration for fastdfs-storage-group1.service is identical to Machine A - simply copy it.
-
Execute the startup script until both fastdfs-storage-group1.service and fastdfs-storage-group3.service are running.
Configuration for Machine C (Group2/Group3)
The configuration process is similar to Machine A, with the following modifications:
- Create directories and copy configuration files:
|
|
- Modify the configuration file for Group2 (
/etc/fdfs/storage-group2.conf). The configuration content for Group3 should remain consistent with Machine B’s settings.
Configuration file for FastDFS Storage group2
$ sudo vim /etc/fdfs/storage-group2.conf
Modified configurations:
group_name=group2 port=33000 # Storage service port (default 23000, modified to 33000) base_path=/data/fastdfs/storage/group2 # Root directory for data and log storage store_path_count=6 store_path0=/data01/fastdfs # First storage directory for group2 store_path1=/data02/fastdfs # Second storage directory for group2 store_path2=/data03/fastdfs # Third storage directory for group2 store_path3=/data04/fastdfs # Fourth storage directory for group2 store_path4=/data05/fastdfs # Fifth storage directory for group2 store_path5=/data06/fastdfs # Sixth storage directory for group2 tracker_server=10.58.10.136:22122 # Tracker server IP and port tracker_server=10.58.10.137:22122 # Tracker server IP and port tracker_server=10.58.10.138:22122 # Tracker server IP and port http.server_port=8888 # HTTP file access port (default 8888, should match nginx configuration)
- The client configuration file is consistent with Machine A, no repetition needed here.
- Create startup services for group2 and group3 (existing templates can be copied directly).
- Execute startup scripts until both fastdfs-storage-group2.service and fastdfs-storage-group3.services are running.
Installing Nginx and the FastDFS Nginx Module
Step 1: In the FastDFS directory, copy the http.conf and mime.types files to the /etc/fdfs directory to enable Nginx access to the Storage service.
|
|
Step 2: Install the FastDFS Nginx module:
|
|
Step 3: Modify the configuration file in fastdfs-nginx-module-1.20/src/config. Locate the ngx_module_incs and CORE_INCS entries and modify them as follows:
|
|
If not modified, the following error will occur during Nginx compilation:
/usr/include/fastdfs/fdfs_define.h:15:27: fatal error: common_define.h: No such file or directory
Step 4: Then extract and install the Nginx service:
$ tar -zxvf nginx-1.12.2.tar.gz $ cd nginx-1.12.2 $ ./configure –prefix=/usr/share/nginx –sbin-path=/usr/sbin/nginx –modules-path=/usr/lib64/nginx/modules –conf-path=/etc/nginx/nginx.conf –error-log-path=/var/log/nginx/error.log –http-log-path=/var/log/nginx/access.log –http-client-body-temp-path=/var/lib/nginx/tmp/client_body –http-proxy-temp-path=/var/lib/nginx/tmp/proxy –http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi –http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi –http-scgi-temp-path=/var/lib/nginx/tmp/scgi –pid-path=/run/nginx.pid –lock-path=/run/lock/subsys/nginx –user=nginx –group=nginx –with-file-aio –with-ipv6 –with-http_auth_request_module –with-http_ssl_module –with-http_v2_module –with-http_realip_module –with-http_addition_module –with-http_xslt_module=dynamic –with-http_image_filter_module=dynamic –with-http_geoip_module=dynamic –with-http_sub_module –with-http_dav_module –with-http_flv_module –with-http_mp4_module –with-http_gunzip_module –with-http_gzip_static_module –with-http_random_index_module –with-http_secure_link_module –with-http_degradation_module –with-http_slice_module –with-http_stub_status_module –with-http_perl_module=dynamic –with-mail=dynamic –with-mail_ssl_module –with-pcre –with-pcre-jit –with-stream=dynamic –with-stream_ssl_module –with-google_perftools_module –with-debug –with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong –param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic’ –with-ld-opt=’-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E’ –add-module=${YOUR_PATH}/fastdfs-nginx-module-1.20/src $ make $ make install
Note: Replace ${YOUR_PATH} in the above with the parent directory of fastdfs-nginx-module-1.20 to ensure the path is correct.
- Modify configuration file /etc/fdfs/mod_fastdfs.conf on Machine A:
connect_timeout=2
network_timeout=30
base_path=/data/fastdfs/ngx_mod
load_fdfs_parameters_from_tracker=true
storage_sync_file_max_delay = 86400
use_storage_id = false
storage_ids_filename = storage_ids.conf
tracker_server=10.58.10.136:22122 # Tracker server IP and port
tracker_server=10.58.10.137:22122 # Tracker server IP and port
tracker_server=10.58.10.138:22122 # Tracker server IP and port
group_name=group1/group2 # Global setting
url_have_group_name = true
log_level=info
log_filename=
response_mode=proxy
if_alias_prefix=
flv_support = true
flv_extension = flv
group_count = 2
[group1]
group_name=group1 # Group-specific
storage_server_port=23000
store_path_count=6
store_path0=/data01/fastdfs
store_path1=/data02/fastdfs
store_path2=/data03/fastdfs
store_path3=/data04/fastdfs
store_path4=/data05/fastdfs
store_path5=/data06/fastdfs
[group2]
group_name=group2
storage_server_port=33000
store_path_count=6
store_path0=/data07/fastdfs
store_path1=/data08/fastdfs
store_path2=/data09/fastdfs
store_path3=/data10/fastdfs
store_path4=/data11/fastdfs
store_path5=/data12/fastdfs
user nginx; worker_processes on; worker_rlimit_nofile 65535;
error_log /var/log/nginx/error.log; pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events { worker_connections 65535; use epoll; accept_mutex off; }
http { log_format main ‘$remote_addr - $remote_user [$time_local] “$request” ’ ‘$status $body_bytes_sent “$http_referer” ’ ‘"$http_user_agent" “$http_x_forwarded_for”’;
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
gzip on;
server_names_hash_bucket_size 128;
client_header_buffer_size 32k;
larger_client_header_buffers 4 32k;
client_max_body_size 300m;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Connection '';
proxy_set_header REMOTE-HOST $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 16k;
proxy_buffers 8 64k;
proxy_busy_buffers_size 128k;
proxy_temp_file_write_size 128k;
proxy_cache_path /data/fastdfs/cache/nginx/proxy_cache levels=1:2
keys_zone=http-cache:200m max_size=1g inactive=30d;
proxy_temp_path /data/fastdfs/cache/nginx/proxy_cache/temp;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
root /usr/share/nginx/html;
# Load default server block configuration
#include /etc/nginx/default.d/*.conf;
location ~ ^/ok(\..*)?$ {
return 200 "OK";
}
location /nginx {
stub_status on;
}
location /healthcheck {
check_status on;
}
location ^~ /group1/ {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
add_header 'Access-Control-Allow-Origin' $http_origin;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header "Access-Control-Allow-Methods" "GET, POST, HEAD, PUT, DELETE, OPTIONS, PATCH";
add_header "Access-Control-Allow-Headers" "Origin, No-Cache, Authorization, X-Requested-With, If-Modified-Since, Pragma, Last-Modified, Cache-Control, Expires, Content-Type";
if ($request_method = 'OPTIONS') {
return 200 'OK';
}
proxy_pass http://fdfs_group1;
expires 30d;
}
location ^~ /group2/ {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
add_header 'Access-Control-Allow-Origin' $http_origin;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header "Access-Control-Allow-Methods" "GET, POST, HEAD, PUT, DELETE, OPTIONS, PATCH";
add_header "Access-Control-Allow-Headers" "Origin, No-Cache, Authorization, X-Requested-With, If-Modified-Since, Pragma, Last-Modified, Cache-Control, Expires, Content-Type";
if ($request_method = 'OPTIONS') {
return 200 'OK';
}
proxy_pass http://fdfs_group2;
expires 30d;
}
location ^~ /group3/ {
proxy_next_upstream http_502 http_504 error timeout invalid_header;
proxy_cache http-cache;
proxy_cache_valid 200 304 12h;
proxy_cache_key $uri$is_args$args;
add_header 'Access-Control-Allow-Origin' $http_origin;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header "Access-Control-Allow-Methods" "GET, POST, HEAD, PUT, DELETE, OPTIONS, PATCH";
add_header "Access-Control-Allow-Headers" "Origin, No-Cache, Authorization, X-Requested-With, If-Modified-Since, Pragma, Last-Modified, Cache-Control, Expires, Content-Type";
if ($request_method = 'OPTIONS') {
return 200 'OK';
}
proxy_pass http://fdfs_group3;
expires 30d;
}
location ~/purge(/.*) {
allow 127.0.0.1;
allow 192.168.1.0/24;
allow 10.58.1.0/24;
deny all;
proxy_cache_purge http-cache $1$is_args$args;
}
}
server {
listen 8888;
server_name localhost;
location /ok.htm {
return 200 "OK";
}
location ~/group[0-9]/ {
ngx_fastdfs_module;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
upstream fdfs_group1 {
server 10.58.10.136:8888 max_fails=0;
server 10.58.10.137:8888 max_fails=0;
keepalive 10240;
check interval=2000 rise=2 fall=3 timeout=1000 type=http default_down=false;
check_http_send "GET /ok.htm HTTP/1.0\r\nConnection:keep-alive\r\n\r\n";
check_keepalive_requests 100;
}
upstream fdfs_group2 {
server 10.58.10.136:8888 max_fails=0;
server 10.58.10.138:8888 max_fails=0;
keepalive 10240;
check interval=2000 rise=2 fall=3 timeout=1000 type=http default_down=false;
check_http_send "GET /ok.htm HTTP/1.0\r\nConnection:keep-alive\r\n\r\n";
check_keepalive_requests 100;
}
upstream fdfs_group3 {
server 10.58.10.137:8888 max_fails=0;
server 10.58.10.138:8888 max_fails=0;
keepalive 10240;
check interval=2000 rise=2 fall=3 timeout=1000 type=http default_down=false;
check_http_send "GET /ok.htm HTTP/1.0\r\nConnection:keep-alive\r\n\r\n";
check_keepalive_requests 100;
}
}
- Start the nginx service:
sudo nginx -c /etc/nginx/nginx.conf
Access http://localhost/ok.htm to verify if it returns a 200 status code with “OK” content. If startup fails, check the nginx logs located at /var/log/nginx/error.log. This is the default error log path for nginx in CentOS. If the error_log configuration has been modified, check the log file specified in the error_log directive.
- The configurations for servers B and C are essentially identical, with completely consistent nginx configuration files. The only required modification is in the /etc/fdfs/mod_fastdfs.conf file: adjust the group_name according to the corresponding storage node configuration. Note that the [groupX] identifiers in square brackets must follow sequential increments starting from 1.
Issue Summary
When encountering issues, first, we troubleshoot service startup failures through console log messages; second, investigate via service logs. In most cases, logs are the most effective troubleshooting tool, bar none.
- Logs for tracker or storage services are stored under the
logsdirectory within the path specified by thebase_pathconfiguration in their respective service configuration files. - For nginx logs, if a custom log path is configured, check the specified directory. By default on CentOS, logs are typically located under
/var/log/nginx/. - Additionally, in this environment, we must also investigate logs for the fastdfs-nginx-module extension. Its logs reside in the directory specified by the
base_pathconfiguration in/etc/fdfs/mod_fastdfs.conf.
Port Configuration for Multiple Groups in a Cluster
- Each
grouprequires a dedicatedstorageservice. Thus, port numbers for storage groups on the same host must not conflict. For example:- Group1: Port
23000 - Group2: Port
33000 - Group3: Port
43000
- Group1: Port
- Groups with the same name cannot exist on the same machine because the tracker synchronizes storage services within the same group, which requires identical port numbers. Therefore, groups with the same name across different hosts must use the same storage port configuration.
Storage Path Configuration for Multiple Groups in a Cluster
- On the same host, storage paths for different groups must be configured separately and should not overlap.
- Across different hosts, storage paths for groups with the same name may differ, but the number of storage nodes and disk capacity should remain consistent.
Configuring Multiple Storage Services on the Same Node
In a 3-node cluster, two groups were deployed on the same machine, with each group maintaining a replica on another node. When starting the services using systemctl start fastdfs-storage-groupx.service (where groupx represents group1, group2, or group3), one group repeatedly failed to start, displaying:
|
|
Resolution: A systemctl daemon-reload is required before starting each storage service.
Example: To start both fastdfs-storage-group1.service and fastdfs-storage-group2.service on Machine A:
- Start
fastdfs-storage-group1.service:
|
|
- Start
fastdfs-storage-group2.service:
|
|
FastDFS Storage Service Configuration
On all storage machines, when modifying /etc/fdfs/mod_fastdfs.conf, adjust the group_name value according to the respective group of each storage. Pay special attention to two configuration items. Below is an example using group2 and group3:
- Global
group_name
Thegroup_namein the global configuration must be separated by “/”, and its value should match thegroup_namein local configurations. For example:
|
|
Steps to Configure and Start Services
- Execute the following commands for group1:
|
|
- Check the status of
fastdfs-storage-group1.service:
|
|
If the status shows active (running), proceed to start fastdfs-storage-group2.service. If not, check logs to troubleshoot:
|
|
- Monitor the status of
fastdfs-storage-group2.serviceuntil it becomes active.
Notes
- You may encounter a warning like
Unknown lvalue 'ExecRestart' in section 'Service'. A known workaround (yum install systemd-*) might not resolve this issue. Contributions to fix this are welcome.
FastDFS Nginx Module Configuration
When modifying /etc/fdfs/mod_fastdfs.conf on storage machines:
- Ensure
group_namevalues are correctly set for each storage group (e.g.,group2andgroup3). - Follow the global/local
group_nameformat as described above.
<group_entries>