前端之家

  1. 首页
  2. 问答

Ceph RGW云同步到Wasabi失败

• 问答

我似乎找不到任何可靠的文档，也似乎找不到能使它正常工作的任何人，因此在这里提出这个问题，我将非常感谢您的帮助。

我遵循以下指南：

https://docs.ceph.com/docs/master/radosgw/cloud-sync-module/ https://docs.ceph.com/docs/master/radosgw/multisite/

我正在尝试设置RGW以将对象同步到Wasabi（S3）以进行异地复制。我有以下安装：

+---------------+  +---------------+
|               |  |               |
| zone: ca-east |  | zone: wasabi  |
|               |  |               |
|  +---------+  |  |  +---------+  |
|  |         |  |  |  |         |  |
|  |  node1  |  |  |  |  node2  |  |
|  |         |  |  |  |         |  |
|  +---------+  |  |  +---------+  |
|               |  |               |
+---------------+  +---------------+

从我对文档的理解来看，似乎每个区域（包括云区域）都需要一个RGW，因此我为云区域创建了一个，并将其配置为与主RGW同步。

它根本不同步，只是处于以下状态：

rgw sync status --rgw-zone=wasabi
          realm 075dcd33-7c67-4870-8154-1ed7672186c3 (company-ceph)
      zonegroup 012eb54f-8d95-4132-85e1-0f7c40bd884d (global)
           zone c0ed3141-a7b6-4e96-8cfb-d0bf0baa2b6a (wasabi)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: 9c2771fe-f00c-4356-9f13-d6fb7900fa77 (ca-east)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

我已经尝试过以我能想到的所有方式更改配置，但是似乎没有什么原因导致node2实际向s3端点（s3.wasabisys.com）发出任何请求。

这是我的配置文件，

区域组：

{
    "id": "012eb54f-8d95-4132-85e1-0f7c40bd884d","name": "global","api_name": "default","is_master": "true","endpoints": [
        "http://ctl.stack.example.net:7480"
    ],"hostnames": [],"hostnames_s3website": [],"master_zone": "9c2771fe-f00c-4356-9f13-d6fb7900fa77","zones": [
        {
            "id": "9c2771fe-f00c-4356-9f13-d6fb7900fa77","name": "ca-east","endpoints": [
                "http://ctl.stack.example.net:7480"
            ],"log_meta": "true","log_data": "true","bucket_index_max_shards": 0,"read_only": "false","tier_type": "","sync_from_all": "true","sync_from": [],"redirect_zone": ""
        },{
            "id": "c0ed3141-a7b6-4e96-8cfb-d0bf0baa2b6a","name": "wasabi","endpoints": [
                "http://node2.stack.example.net:7480"
            ],"read_only": "true","tier_type": "cloud","redirect_zone": ""
        }
    ],"placement_targets": [
        {
            "name": "default-placement","tags": [],"storage_classes": [
                "STANDARD"
            ]
        }
    ],"default_placement": "default-placement","realm_id": "075dcd33-7c67-4870-8154-1ed7672186c3"
}

区域（东部）：

{
    "id": "9c2771fe-f00c-4356-9f13-d6fb7900fa77","domain_root": "default.rgw.meta:root","control_pool": "default.rgw.control","gc_pool": "default.rgw.log:gc","lc_pool": "default.rgw.log:lc","log_pool": "default.rgw.log","intent_log_pool": "default.rgw.log:intent","usage_log_pool": "default.rgw.log:usage","reshard_pool": "default.rgw.log:reshard","user_keys_pool": "default.rgw.meta:users.keys","user_email_pool": "default.rgw.meta:users.email","user_swift_pool": "default.rgw.meta:users.swift","user_uid_pool": "default.rgw.meta:users.uid","otp_pool": "default.rgw.otp","system_key": {
        "access_key": "SOMEKEY","secret_key": "SOMEKEY"
    },"placement_pools": [
        {
            "key": "default-placement","val": {
                "index_pool": "default.rgw.buckets.index","storage_classes": {
                    "STANDARD": {
                        "data_pool": "default.rgw.buckets.data"
                    }
                },"data_extra_pool": "default.rgw.buckets.non-ec","index_type": 0
            }
        }
    ],"metadata_heap": "","realm_id": "075dcd33-7c67-4870-8154-1ed7672186c3"
}

区域（芥末）：

{
    "id": "c0ed3141-a7b6-4e96-8cfb-d0bf0baa2b6a","domain_root": "wasabi.rgw.meta:root","control_pool": "wasabi.rgw.control","gc_pool": "wasabi.rgw.log:gc","lc_pool": "wasabi.rgw.log:lc","log_pool": "wasabi.rgw.log","intent_log_pool": "wasabi.rgw.log:intent","usage_log_pool": "wasabi.rgw.log:usage","reshard_pool": "wasabi.rgw.log:reshard","user_keys_pool": "wasabi.rgw.meta:users.keys","user_email_pool": "wasabi.rgw.meta:users.email","user_swift_pool": "wasabi.rgw.meta:users.swift","user_uid_pool": "wasabi.rgw.meta:users.uid","otp_pool": "wasabi.rgw.otp","val": {
                "index_pool": "wasabi.rgw.buckets.index","storage_classes": {
                    "STANDARD": {
                        "data_pool": "wasabi.rgw.buckets.data"
                    }
                },"data_extra_pool": "wasabi.rgw.buckets.non-ec","tier_config": {
        "acls": [
            {
                "dest_id": "someuser@example.net","source_id": "git@example.net","type": "uri"
            }
        ],"connection": {
            "access_key": "SOMEKEY","endpoint": "https://s3.wasabisys.com:443","host_style": "virtual","secret": "SOMEKEY"
        },"target_path": "rgwx-${zone}-${sid}/${owner}/${bucket}"
    },"realm_id": "075dcd33-7c67-4870-8154-1ed7672186c3"
}

ceph.conf（节点2）：

[global]
fsid = d707fc08-1256-490c-bfff-155561721739
mon initial members = node1
mon host = 172.16.20.20
public network = 172.16.20.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1

[mgr]
mgr_modules = dashboard

[mds.node1]
host = node1

[client.images]
keyring = /etc/ceph/ceph.client.images.keyring

[client.volumes]
keyring = /etc/ceph/ceph.client.volumes.keyring

[client]
    rbd cache = true
    rbd cache writethrough until flush = true
    log file = /var/log/qemu/qemu-guest-$pid.log
    rbd concurrent management ops = 20

[client.rgw.node2]
host = node2
#rgw dns name = node2.stack.example.net
rgw_zone = wasabi
# Keystone information
rgw keystone url = https://ctl.stack.example.net:5000
rgw keystone api version = 3
rgw keystone admin user = admin
rgw keystone admin password = thebestpassword
rgw keystone admin domain = default
rgw keystone admin project = admin
rgw keystone accepted roles =  _member_,Member,admin,swiftoperator
rgw keystone implicit tenants = false
rgw keystone token cache size = 10
rgw keystone revocation interval = 0
rgw swift account in url = true
rgw s3 auth use keystone = false

debug ms = 1
debug rgw = 20
log file = /var/log/ceph/ceph-rgw.log

启用调试模式时，我会看到与I / O错误相关的有趣日志，但并不确定它们所指的什么I / O问题，因为系统本身似乎没有任何I / O问题。> 

2020-01-03 16:36:05.397 7fe2c6270700 20 run: stack=0x55ee7dbdafc0 is io blocked
2020-01-03 16:36:05.438 7fe2c6270700 20 run: stack=0x55ee7dbdafc0 is io blocked
2020-01-03 16:36:05.441 7fe2c6270700 20 run: stack=0x55ee7deea240 is io blocked
2020-01-03 16:36:05.441 7fe2c6270700 20 run: stack=0x55ee7deea240 is io blocked
2020-01-03 16:36:05.482 7fe2c6270700 20 run: stack=0x55ee7deea240 is io blocked

rgw日志（node2）：（仅供参考：ctl.stack与node1.stack相同）

2020-01-03 16:36:44.915 7fe2c6a71700 20 link_request req_data=0x55ee7e34b200 req_data->id=4980,curl_handle=0x55ee7e89ade0
2020-01-03 16:36:44.917 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.917 7fe2c6a71700 10 received header:HTTP/1.1 200 OK
2020-01-03 16:36:44.917 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.917 7fe2c6a71700 10 received header:x-amz-request-id: tx000000000000000005247-005e0fb3ec-1de72d-ca-east
2020-01-03 16:36:44.917 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.917 7fe2c6a71700 10 received header:Content-Length: 44
2020-01-03 16:36:44.917 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.917 7fe2c6a71700 10 received header:Date: Fri,03 Jan 2020 21:36:44 GMT
2020-01-03 16:36:44.917 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.917 7fe2c6a71700 10 received header:Connection: Keep-Alive
2020-01-03 16:36:44.917 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.917 7fe2c6a71700 10 received header:
2020-01-03 16:36:44.957 7fe2c6270700 20 cr:s=0x55ee7da2be60:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:44.957 7fe2c6270700 20 meta sync: operate: shard_id=45: receiving rest response
2020-01-03 16:36:44.957 7fe2c6270700 20 meta sync: remote mdlog,shard_id=45 num of shard entries: 0
2020-01-03 16:36:44.957 7fe2c6270700 20 cr:s=0x55ee7da2be60:op=0x55ee7e890000:18RGWMetaSyncShardCR: operate()
2020-01-03 16:36:44.957 7fe2c6270700 20 meta sync: incremental_sync:1796: shard_id=45 mdlog_marker= max_marker= sync_marker.marker= period_marker=
2020-01-03 16:36:44.957 7fe2c6270700 20 run: stack=0x55ee7da2be60 is io blocked
2020-01-03 16:36:44.972 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e499800:18RGWMetaSyncShardCR: operate()
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: incremental_sync:1729: shard_id=6 mdlog_marker= sync_marker.marker= period_marker=
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: incremental_sync:1737: shard_id=6 syncing mdlog for shard_id=6
2020-01-03 16:36:44.972 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: operate: shard_id=6: init request
2020-01-03 16:36:44.972 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: operate: shard_id=6: reading shard status
2020-01-03 16:36:44.972 7fe2c6270700  1 -- 172.16.20.21:0/3527532116 --> [v2:172.16.20.24:6832/3451,v1:172.16.20.24:6833/3451] -- osd_op(unknown.0.0:13776 23.3 23:caf68780:::meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.6:head [call log.info] snapc 0=[] ondisk+read+known_if_redirected e2345) v8 -- 0x55ee7e710dc0 con 0x55ee7db30c00
2020-01-03 16:36:44.972 7fe2c6270700 20 run: stack=0x55ee7dbdbd40 is io blocked
2020-01-03 16:36:44.972 7fe2e6d3b700  1 -- 172.16.20.21:0/3527532116 <== osd.13 v2:172.16.20.24:6832/3451 1825 ==== osd_op_reply(13776 meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.6 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 191+0+0 (crc 0 0 0) 0x55ee7ec92780 con 0x55ee7db30c00
2020-01-03 16:36:44.972 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: operate: shard_id=6: reading shard status complete
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: shard_id=6 marker= last_update=1969-12-31 19:00:00.000000s
2020-01-03 16:36:44.972 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:44.972 7fe2c6270700 20 meta sync: operate: shard_id=6: sending rest request
2020-01-03 16:36:44.972 7fe2c6270700 20 > HTTP_DATE -> Fri,03 Jan 2020 21:36:44 +0000
2020-01-03 16:36:44.972 7fe2c6270700 10 get_canon_resource(): dest=/admin/log
2020-01-03 16:36:44.972 7fe2c6270700 10 generated canonical header: GET


Fri,03 Jan 2020 21:36:44 +0000
/admin/log
2020-01-03 16:36:44.972 7fe2c6270700 15 generated auth header: AWS BLAHBLAHBLAH
2020-01-03 16:36:44.972 7fe2c6270700 20 sending request to http://ctl.stack.example.net:7480/admin/log?type=metadata&id=6&period=0dc0d5b4-81bf-416c-b096-4e9f253caf53&max-entries=100&&rgwx-zonegroup=012eb54f-8d95-4132-85e1-0f7c40bd884d
2020-01-03 16:36:44.972 7fe2c6270700 20 register_request mgr=0x55ee7db30938 req_data->id=4981,curl_handle=0x55ee7e89ade0
2020-01-03 16:36:44.972 7fe2c6270700 20 run: stack=0x55ee7dbdbd40 is io blocked
2020-01-03 16:36:44.972 7fe2c6a71700 20 link_request req_data=0x55ee7e34b200 req_data->id=4981,curl_handle=0x55ee7e89ade0
2020-01-03 16:36:44.974 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.974 7fe2c6a71700 10 received header:HTTP/1.1 200 OK
2020-01-03 16:36:44.974 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.974 7fe2c6a71700 10 received header:x-amz-request-id: tx000000000000000005248-005e0fb3ec-1de72d-ca-east
2020-01-03 16:36:44.974 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.974 7fe2c6a71700 10 received header:Content-Length: 44
2020-01-03 16:36:44.974 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.974 7fe2c6a71700 10 received header:Date: Fri,03 Jan 2020 21:36:44 GMT
2020-01-03 16:36:44.974 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.974 7fe2c6a71700 10 received header:Connection: Keep-Alive
2020-01-03 16:36:44.974 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:44.974 7fe2c6a71700 10 received header:
2020-01-03 16:36:45.014 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.014 7fe2c6270700 20 meta sync: operate: shard_id=6: receiving rest response
2020-01-03 16:36:45.014 7fe2c6270700 20 meta sync: remote mdlog,shard_id=6 num of shard entries: 0
2020-01-03 16:36:45.014 7fe2c6270700 20 cr:s=0x55ee7dbdbd40:op=0x55ee7e499800:18RGWMetaSyncShardCR: operate()
2020-01-03 16:36:45.014 7fe2c6270700 20 meta sync: incremental_sync:1796: shard_id=6 mdlog_marker= max_marker= sync_marker.marker= period_marker=
2020-01-03 16:36:45.014 7fe2c6270700 20 run: stack=0x55ee7dbdbd40 is io blocked
2020-01-03 16:36:45.023 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e49d000:18RGWMetaSyncShardCR: operate()
2020-01-03 16:36:45.023 7fe2c6270700 20 meta sync: incremental_sync:1729: shard_id=19 mdlog_marker= sync_marker.marker= period_marker=
2020-01-03 16:36:45.023 7fe2c6270700 20 meta sync: incremental_sync:1737: shard_id=19 syncing mdlog for shard_id=19
2020-01-03 16:36:45.023 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.023 7fe2c6270700 20 meta sync: operate: shard_id=19: init request
2020-01-03 16:36:45.023 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.023 7fe2c6270700 20 meta sync: operate: shard_id=19: reading shard status
2020-01-03 16:36:45.023 7fe2c6270700  1 -- 172.16.20.21:0/3527532116 --> [v2:172.16.20.23:6815/50448,v1:172.16.20.23:6816/50448] -- osd_op(unknown.0.0:13777 23.1 23:91499bbe:::meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.19:head [call log.info] snapc 0=[] ondisk+read+known_if_redirected e2345) v8 -- 0x55ee7ea9a840 con 0x55ee7db31000
2020-01-03 16:36:45.023 7fe2c6270700 20 run: stack=0x55ee7da2a120 is io blocked
2020-01-03 16:36:45.024 7fe2e5d39700  1 -- 172.16.20.21:0/3527532116 <== osd.2 v2:172.16.20.23:6815/50448 1850 ==== osd_op_reply(13777 meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.19 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 192+0+0 (crc 0 0 0) 0x55ee7ec92280 con 0x55ee7db31000
2020-01-03 16:36:45.024 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.024 7fe2c6270700 20 meta sync: operate: shard_id=19: reading shard status complete
2020-01-03 16:36:45.024 7fe2c6270700 20 meta sync: shard_id=19 marker= last_update=1969-12-31 19:00:00.000000s
2020-01-03 16:36:45.024 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.024 7fe2c6270700 20 meta sync: operate: shard_id=19: sending rest request
2020-01-03 16:36:45.024 7fe2c6270700 20 > HTTP_DATE -> Fri,03 Jan 2020 21:36:45 +0000
2020-01-03 16:36:45.024 7fe2c6270700 10 get_canon_resource(): dest=/admin/log
2020-01-03 16:36:45.024 7fe2c6270700 10 generated canonical header: GET


Fri,03 Jan 2020 21:36:45 +0000
/admin/log
2020-01-03 16:36:45.024 7fe2c6270700 15 generated auth header: AWS BLAHBLAHBLAH
2020-01-03 16:36:45.024 7fe2c6270700 20 sending request to http://ctl.stack.example.net:7480/admin/log?type=metadata&id=19&period=0dc0d5b4-81bf-416c-b096-4e9f253caf53&max-entries=100&&rgwx-zonegroup=012eb54f-8d95-4132-85e1-0f7c40bd884d
2020-01-03 16:36:45.024 7fe2c6270700 20 register_request mgr=0x55ee7db30938 req_data->id=4982,curl_handle=0x55ee7e89ade0
2020-01-03 16:36:45.024 7fe2c6270700 20 run: stack=0x55ee7da2a120 is io blocked
2020-01-03 16:36:45.024 7fe2c6a71700 20 link_request req_data=0x55ee7e34b200 req_data->id=4982,curl_handle=0x55ee7e89ade0
2020-01-03 16:36:45.026 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:45.026 7fe2c6a71700 10 received header:HTTP/1.1 200 OK
2020-01-03 16:36:45.026 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:45.026 7fe2c6a71700 10 received header:x-amz-request-id: tx000000000000000005249-005e0fb3ed-1de72d-ca-east
2020-01-03 16:36:45.026 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:45.026 7fe2c6a71700 10 received header:Content-Length: 44
2020-01-03 16:36:45.026 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:45.026 7fe2c6a71700 10 received header:Date: Fri,03 Jan 2020 21:36:45 GMT
2020-01-03 16:36:45.026 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:45.026 7fe2c6a71700 10 received header:Connection: Keep-Alive
2020-01-03 16:36:45.026 7fe2c6a71700 10 receive_http_header
2020-01-03 16:36:45.026 7fe2c6a71700 10 received header:
2020-01-03 16:36:45.066 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.066 7fe2c6270700 20 meta sync: operate: shard_id=19: receiving rest response
2020-01-03 16:36:45.066 7fe2c6270700 20 meta sync: remote mdlog,shard_id=19 num of shard entries: 0
2020-01-03 16:36:45.066 7fe2c6270700 20 cr:s=0x55ee7da2a120:op=0x55ee7e49d000:18RGWMetaSyncShardCR: operate()
2020-01-03 16:36:45.066 7fe2c6270700 20 meta sync: incremental_sync:1796: shard_id=19 mdlog_marker= max_marker= sync_marker.marker= period_marker=
2020-01-03 16:36:45.066 7fe2c6270700 20 run: stack=0x55ee7da2a120 is io blocked
2020-01-03 16:36:45.073 7fe2c6270700 20 cr:s=0x55ee7dbdb680:op=0x55ee7e4ba000:18RGWMetaSyncShardCR: operate()
2020-01-03 16:36:45.073 7fe2c6270700 20 meta sync: incremental_sync:1729: shard_id=9 mdlog_marker= sync_marker.marker= period_marker=
2020-01-03 16:36:45.073 7fe2c6270700 20 meta sync: incremental_sync:1737: shard_id=9 syncing mdlog for shard_id=9
2020-01-03 16:36:45.073 7fe2c6270700 20 cr:s=0x55ee7dbdb680:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.073 7fe2c6270700 20 meta sync: operate: shard_id=9: init request
2020-01-03 16:36:45.073 7fe2c6270700 20 cr:s=0x55ee7dbdb680:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.073 7fe2c6270700 20 meta sync: operate: shard_id=9: reading shard status
2020-01-03 16:36:45.073 7fe2c6270700  1 -- 172.16.20.21:0/3527532116 --> [v2:172.16.20.23:6824/50455,v1:172.16.20.23:6825/50455] -- osd_op(unknown.0.0:13778 23.2 23:587a49fc:::meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.9:head [call log.info] snapc 0=[] ondisk+read+known_if_redirected e2345) v8 -- 0x55ee7eb84580 con 0x55ee7db31400
2020-01-03 16:36:45.073 7fe2c6270700 20 run: stack=0x55ee7dbdb680 is io blocked
2020-01-03 16:36:45.074 7fe2e653a700  1 -- 172.16.20.21:0/3527532116 <== osd.7 v2:172.16.20.23:6824/50455 1740 ==== osd_op_reply(13778 meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.9 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 191+0+0 (crc 0 0 0) 0x55ee7e798a00 con 0x55ee7db31400
2020-01-03 16:36:45.074 7fe2c6270700 20 cr:s=0x55ee7dbdb680:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.074 7fe2c6270700 20 meta sync: operate: shard_id=9: reading shard status complete
2020-01-03 16:36:45.074 7fe2c6270700 20 meta sync: shard_id=9 marker= last_update=1969-12-31 19:00:00.000000s
2020-01-03 16:36:45.074 7fe2c6270700 20 cr:s=0x55ee7dbdb680:op=0x55ee7e345e00:24RGWCloneMetaLogCoroutine: operate()
2020-01-03 16:36:45.074 7fe2c6270700 20 meta sync: operate: shard_id=9: sending rest request
2020-01-03 16:36:45.074 7fe2c6270700 20 > HTTP_DATE -> Fri,03 Jan 2020 21:36:45 +0000
2020-01-03 16:36:45.074 7fe2c6270700 10 get_canon_resource(): dest=/admin/log
2020-01-03 16:36:45.074 7fe2c6270700 10 generated canonical header: GET

节点之间似乎有很多请求，但是没有什么要向云服务发送。这真的让我认为存在一个奇怪的配置错误问题...

node1 rgw日志：

Fri,03 Jan 2020 21:42:05 +0000
/admin/log
2020-01-03 16:42:05.672 7f8551077700 15 req 24142 0.000s :list_metadata_log string_to_sign=GET


Fri,03 Jan 2020 21:42:05 +0000
/admin/log
2020-01-03 16:42:05.672 7f8551077700 15 req 24142 0.000s :list_metadata_log server signature=SIG
2020-01-03 16:42:05.672 7f8551077700 15 req 24142 0.000s :list_metadata_log client signature=SIG
2020-01-03 16:42:05.672 7f8551077700 15 req 24142 0.000s :list_metadata_log compare=0
2020-01-03 16:42:05.672 7f8551077700 20 req 24142 0.000s :list_metadata_log rgw::auth::s3::LocalEngine granted access
2020-01-03 16:42:05.672 7f8551077700 20 req 24142 0.000s :list_metadata_log rgw::auth::s3::AWSAuthStrategy granted access
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log normalizing buckets and tenants
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log init permissions
2020-01-03 16:42:05.672 7f8551077700 20 get_system_obj_state: rctx=0x561ac8f83400 obj=default.rgw.meta:users.uid:sync-user state=0x561ac8cfaac0 s->prefetch_data=0
2020-01-03 16:42:05.672 7f8551077700 10 cache get: name=default.rgw.meta+users.uid+sync-user : hit (requested=0x6,cached=0x17)
2020-01-03 16:42:05.672 7f8551077700 20 get_system_obj_state: s->obj_tag was set empty
2020-01-03 16:42:05.672 7f8551077700 20 Read xattr: user.rgw.idtag
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log recalculating target
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log reading permissions
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log init op
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log verifying op mask
2020-01-03 16:42:05.672 7f8551077700 20 req 24142 0.000s :list_metadata_log required_mask= 0 user.op_mask=7
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log verifying op permissions
2020-01-03 16:42:05.672 7f8551077700  2 overriding permissions due to system operation
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log verifying op params
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log pre-executing
2020-01-03 16:42:05.672 7f8551077700  2 req 24142 0.000s :list_metadata_log executing
2020-01-03 16:42:05.672 7f8551077700  1 -- 172.16.20.20:0/2649741282 --> [v2:172.16.20.24:6832/3451,v1:172.16.20.24:6833/3451] -- osd_op(unknown.0.0:26809 15.4 15:367d3812:::meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.52:head [call log.list] snapc 0=[] ondisk+read+known_if_redirected e2345) v8 -- 0x561ac943a580 con 0x561ac7101000
2020-01-03 16:42:05.673 7f8668d30700  1 -- 172.16.20.20:0/2649741282 <== osd.13 v2:172.16.20.24:6832/3451 3411 ==== osd_op_reply(26809 meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.52 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 192+0+0 (crc 0 0 0) 0x561ac954c280 con 0x561ac7101000
2020-01-03 16:42:05.673 7f8551077700  2 req 24142 0.001s :list_metadata_log completing
2020-01-03 16:42:05.673 7f8551077700  2 req 24142 0.001s :list_metadata_log op status=0
2020-01-03 16:42:05.673 7f8551077700  2 req 24142 0.001s :list_metadata_log http status=200
2020-01-03 16:42:05.673 7f8551077700  1 ====== req done req=0x561ac8f84910 op status=0 http_status=200 latency=0.00100002s ======
2020-01-03 16:42:05.716 7f863fa54700 20 HTTP_accEPT=*/*
2020-01-03 16:42:05.716 7f863fa54700 20 HTTP_AUTHORIZATION=AWS SOMEKEY:SIG
2020-01-03 16:42:05.716 7f863fa54700 20 HTTP_DATE=Fri,03 Jan 2020 21:42:05 +0000
2020-01-03 16:42:05.716 7f863fa54700 20 HTTP_HOST=ctl.stack.example.net:7480
2020-01-03 16:42:05.716 7f863fa54700 20 HTTP_VERSION=1.1
2020-01-03 16:42:05.716 7f863fa54700 20 QUERY_STRING=type=metadata&id=61&period=0dc0d5b4-81bf-416c-b096-4e9f253caf53&max-entries=100&&rgwx-zonegroup=012eb54f-8d95-4132-85e1-0f7c40bd884d
2020-01-03 16:42:05.716 7f863fa54700 20 REMOTE_ADDR=172.16.20.21
2020-01-03 16:42:05.716 7f863fa54700 20 REQUEST_METHOD=GET
2020-01-03 16:42:05.716 7f863fa54700 20 REQUEST_URI=/admin/log
2020-01-03 16:42:05.716 7f863fa54700 20 SCRIPT_URI=/admin/log
2020-01-03 16:42:05.716 7f863fa54700 20 SERVER_PORT=7480
2020-01-03 16:42:05.716 7f863fa54700  1 ====== starting new request req=0x561ac8f84910 =====
2020-01-03 16:42:05.716 7f863fa54700  2 req 24143 0.000s initializing for trans_id = tx000000000000000005e4f-005e0fb52d-1de72d-ca-east
2020-01-03 16:42:05.716 7f863fa54700 10 rgw api priority: s3=7 s3website=6
2020-01-03 16:42:05.716 7f863fa54700 10 host=ctl.stack.example.net
2020-01-03 16:42:05.716 7f863fa54700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2020-01-03 16:42:05.716 7f863fa54700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/admin/log
2020-01-03 16:42:05.716 7f863fa54700 10 handler=14RGWHandler_Log
2020-01-03 16:42:05.716 7f863fa54700  2 req 24143 0.000s getting op 0
2020-01-03 16:42:05.716 7f863fa54700 10 req 24143 0.000s :list_metadata_log scheduling with dmclock client=0 cost=1
2020-01-03 16:42:05.716 7f863fa54700 10 op=16RGWOp_MDLog_List
2020-01-03 16:42:05.716 7f863fa54700  2 req 24143 0.000s :list_metadata_log verifying requester
2020-01-03 16:42:05.716 7f863fa54700 20 req 24143 0.000s :list_metadata_log rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy
2020-01-03 16:42:05.716 7f863fa54700 20 req 24143 0.000s :list_metadata_log rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
2020-01-03 16:42:05.716 7f863fa54700 20 req 24143 0.000s :list_metadata_log rgw::auth::s3::S3AnonymousEngine denied with reason=-1
2020-01-03 16:42:05.716 7f863fa54700 20 req 24143 0.000s :list_metadata_log rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
2020-01-03 16:42:05.716 7f863fa54700 10 get_canon_resource(): dest=/admin/log
2020-01-03 16:42:05.716 7f863fa54700 10 string_to_sign:
GET
Fri,03 Jan 2020 21:42:05 +0000
/admin/log
2020-01-03 16:42:05.716 7f863fa54700 15 req 24143 0.000s :list_metadata_log string_to_sign=GET

不确定找不到该文件/目录的内容。经过研究后，似乎与其中一个节点上不存在mdl​​og有关？ ：

2020-01-03 16:42:05.895 7f8668d30700  1 -- 172.16.20.20:0/2649741282 <== osd.13 v2:172.16.20.24:6832/3451 3413 ==== osd_op_reply(26813 meta.log.0dc0d5b4-81bf-416c-b096-4e9f253caf53.48 [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 192+0+0 (crc 0 0 0) 0x561ac954c280 con 0x561ac7101000

我不确定是在服务器故障还是在ceph邮件列表上发布它。

jiguanni 回答：Ceph RGW云同步到Wasabi失败
暂时没有好的解决方案，如果你有好的解决方案，请发邮件至：iooj@foxmail.com
cephradosgw
本文链接：https://www.f2er.com/2822400.html

大家都在问