前端之家

  1. 首页
  2. 问答

OpenEBS cstore卷在删除和覆盖时非常慢

• 问答

我有一个OpenEBS设置，带有3个数据节点和一个默认的cstore存储类。 创建文件效果很好：

time dd if=/dev/urandom of=a.log bs=1M count=100
real    0m 0.53s

我可以删除文件并在相同的时间再次创建它。 但是，当我重写该文件时，它会花费很多时间：

time dd if=/dev/urandom of=a.log bs=1M count=100
104857600 bytes (100.0MB) copied,0.596577 seconds,167.6MB/s
real    0m 0.59s

time dd if=/dev/urandom of=a.log bs=1M count=100
104857600 bytes (100.0MB) copied,16.621222 seconds,6.0MB/s
real    0m 16.62s

time dd if=/dev/urandom of=a.log bs=1M count=100
104857600 bytes (100.0MB) copied,19.621924 seconds,5.1MB/s
real    0m 19.62s

当我删除a.log并再次写入时，〜167MS / s又回来了。仅写入现有文件需要花费大量时间。

我的问题是我认为这就是为什么我的某些应用程序（例如数据库）太慢的原因。在mysql中创建表已接管了7sek。

这是我的测试集群的规格：

# Create the OpenEBS namespace
apiVersion: v1
kind: Namespace
metadata:
  name: openebs
---
# Create Maya Service account
apiVersion: v1
kind: Serviceaccount
metadata:
  name: openebs-maya-operator
  namespace: openebs
---
# Define Role that allows operations on K8s pods/deployments
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: openebs-maya-operator
rules:
- apiGroups: ["*"]
  resources: ["nodes","nodes/proxy"]
  verbs: ["*"]
- apiGroups: ["*"]
  resources: ["namespaces","services","pods","pods/exec","deployments","replicationcontrollers","replicasets","events","endpoints","configmaps","secrets","jobs","cronjobs"]
  verbs: ["*"]
- apiGroups: ["*"]
  resources: ["statefulsets","daemonsets"]
  verbs: ["*"]
- apiGroups: ["*"]
  resources: ["resourcequotas","limitranges"]
  verbs: ["list","watch"]
- apiGroups: ["*"]
  resources: ["ingresses","horizontalpodautoscalers","verticalpodautoscalers","poddisruptionbudgets","certificatesigningrequests"]
  verbs: ["list","watch"]
- apiGroups: ["*"]
  resources: ["storageclasses","persistentvolumeclaims","persistentvolumes"]
  verbs: ["*"]
- apiGroups: ["volumesnapshot.external-storage.k8s.io"]
  resources: ["volumesnapshots","volumesnapshotdatas"]
  verbs: ["get","list","watch","create","update","patch","delete"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: [ "get","delete","patch"]
- apiGroups: ["*"]
  resources: [ "disks","blockdevices","blockdeviceclaims"]
  verbs: ["*" ]
- apiGroups: ["*"]
  resources: [ "cstorpoolclusters","storagepoolclaims","storagepoolclaims/finalizers","cstorpoolclusters/finalizers","storagepools"]
  verbs: ["*" ]
- apiGroups: ["*"]
  resources: [ "castemplates","runtasks"]
  verbs: ["*" ]
- apiGroups: ["*"]
  resources: [ "cstorpools","cstorpools/finalizers","cstorvolumereplicas","cstorvolumes","cstorvolumeclaims"]
  verbs: ["*" ]
- apiGroups: ["*"]
  resources: [ "cstorpoolinstances","cstorpoolinstances/finalizers"]
  verbs: ["*" ]
- apiGroups: ["*"]
  resources: [ "cstorbackups","cstorrestores","cstorcompletedbackups"]
  verbs: ["*" ]
- apiGroups: ["coordination.k8s.io"]
  resources: ["leases"]
  verbs: ["get","create"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
- apiGroups: ["*"]
  resources: [ "upgradetasks"]
  verbs: ["*" ]
---
# Bind the Service account with the Role Privileges.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: openebs-maya-operator
subjects:
- kind: Serviceaccount
  name: openebs-maya-operator
  namespace: openebs
- kind: User
  name: system:serviceaccount:default:default
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: openebs-maya-operator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: maya-apiserver
  namespace: openebs
  labels:
    name: maya-apiserver
    openebs.io/component-name: maya-apiserver
    openebs.io/version: 1.3.0
spec:
  selector:
    matchLabels:
      name: maya-apiserver
      openebs.io/component-name: maya-apiserver
  replicas: 1
  strategy:
    type: Recreate
    rollingUpdate: null
  template:
    metadata:
      labels:
        name: maya-apiserver
        openebs.io/component-name: maya-apiserver
        openebs.io/version: 1.3.0
    spec:
      serviceaccountName: openebs-maya-operator
      containers:
      - name: maya-apiserver
        imagePullPolicy: IfNotPresent
        image: quay.io/openebs/m-apiserver:1.3.0
        ports:
        - containerPort: 5656
        env:
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: OPENEBS_SERVICE_accOUNT
          valueFrom:
            fieldRef:
              fieldPath: spec.serviceaccountName
        - name: OPENEBS_MAYA_pod_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: OPENEBS_IO_CREATE_DEFAULT_STORAGE_CONFIG
          value: "true"
        - name: OPENEBS_IO_INSTALL_DEFAULT_CSTOR_SPARSE_POOL
          value: "true"
        - name: OPENEBS_IO_JIVA_CONTROLLER_IMAGE
          value: "quay.io/openebs/jiva:1.3.0"
        - name: OPENEBS_IO_JIVA_REPLICA_IMAGE
          value: "quay.io/openebs/jiva:1.3.0"
        - name: OPENEBS_IO_JIVA_REPLICA_COUNT
          value: "1"
        - name: OPENEBS_IO_CSTOR_TARGET_IMAGE
          value: "quay.io/openebs/cstor-istgt:1.3.0"
        - name: OPENEBS_IO_CSTOR_POOL_IMAGE
          value: "quay.io/openebs/cstor-pool:1.3.0"
        - name: OPENEBS_IO_CSTOR_POOL_MGMT_IMAGE
          value: "quay.io/openebs/cstor-pool-mgmt:1.3.0"
        - name: OPENEBS_IO_CSTOR_VOLUME_MGMT_IMAGE
          value: "quay.io/openebs/cstor-volume-mgmt:1.3.0"
        - name: OPENEBS_IO_VOLUME_MONITOR_IMAGE
          value: "quay.io/openebs/m-exporter:1.3.0"
        - name: OPENEBS_IO_CSTOR_POOL_EXPORTER_IMAGE
          value: "quay.io/openebs/m-exporter:1.3.0"
        - name: OPENEBS_IO_ENABLE_ANALYTICS
          value: "false"
        - name: OPENEBS_IO_INSTALLER_TYPE
          value: "openebs-operator"
        livenessProbe:
          exec:
            command:
            - /usr/local/bin/mayactl
            - version
          initialDelaySeconds: 30
          periodSeconds: 60
        readinessProbe:
          exec:
            command:
            - /usr/local/bin/mayactl
            - version
          initialDelaySeconds: 30
          periodSeconds: 60
---
apiVersion: v1
kind: Service
metadata:
  name: maya-apiserver-service
  namespace: openebs
  labels:
    openebs.io/component-name: maya-apiserver-svc
spec:
  ports:
  - name: api
    port: 5656
    protocol: TCP
    targetPort: 5656
  selector:
    name: maya-apiserver
  sessionAffinity: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openebs-provisioner
  namespace: openebs
  labels:
    name: openebs-provisioner
    openebs.io/component-name: openebs-provisioner
    openebs.io/version: 1.3.0
spec:
  selector:
    matchLabels:
      name: openebs-provisioner
      openebs.io/component-name: openebs-provisioner
  replicas: 1
  strategy:
    type: Recreate
    rollingUpdate: null
  template:
    metadata:
      labels:
        name: openebs-provisioner
        openebs.io/component-name: openebs-provisioner
        openebs.io/version: 1.3.0
    spec:
      serviceaccountName: openebs-maya-operator
      containers:
      - name: openebs-provisioner
        imagePullPolicy: IfNotPresent
        image: quay.io/openebs/openebs-k8s-provisioner:1.3.0
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodename
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        livenessProbe:
          exec:
            command:
            - pgrep
            - ".*openebs"
          initialDelaySeconds: 30
          periodSeconds: 60
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openebs-snapshot-operator
  namespace: openebs
  labels:
    name: openebs-snapshot-operator
    openebs.io/component-name: openebs-snapshot-operator
    openebs.io/version: 1.3.0
spec:
  selector:
    matchLabels:
      name: openebs-snapshot-operator
      openebs.io/component-name: openebs-snapshot-operator
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: openebs-snapshot-operator
        openebs.io/component-name: openebs-snapshot-operator
        openebs.io/version: 1.3.0
    spec:
      serviceaccountName: openebs-maya-operator
      containers:
        - name: snapshot-controller
          image: quay.io/openebs/snapshot-controller:1.3.0
          imagePullPolicy: IfNotPresent
          env:
          - name: OPENEBS_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          livenessProbe:
            exec:
              command:
              - pgrep
              - ".*controller"
            initialDelaySeconds: 30
            periodSeconds: 60
        - name: snapshot-provisioner
          image: quay.io/openebs/snapshot-provisioner:1.3.0
          imagePullPolicy: IfNotPresent
          env:
          - name: OPENEBS_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          livenessProbe:
            exec:
              command:
              - pgrep
              - ".*provisioner"
            initialDelaySeconds: 30
            periodSeconds: 60
---
# This is the node-disk-manager related config.
# It can be used to customize the disks probes and filters
apiVersion: v1
kind: ConfigMap
metadata:
  name: openebs-ndm-config
  namespace: openebs
  labels:
    openebs.io/component-name: ndm-config
data:
  node-disk-manager.config: |
    probeconfigs:
      - key: udev-probe
        name: udev probe
        state: true
      - key: seachest-probe
        name: seachest probe
        state: false
      - key: smart-probe
        name: smart probe
        state: true
    filterconfigs:
      - key: os-disk-exclude-filter
        name: os disk exclude filter
        state: true
        exclude: "/,/etc/hosts,/boot"
      - key: vendor-filter
        name: vendor filter
        state: true
        include: ""
        exclude: "CLOUDBYT,OpenEBS"
      - key: path-filter
        name: path filter
        state: true
        include: ""
        exclude: "loop,/dev/fd0,/dev/sr0,/dev/ram,/dev/dm-,/dev/md"
---
apiVersion: apps/v1
kind: daemonset
metadata:
  name: openebs-ndm
  namespace: openebs
  labels:
    name: openebs-ndm
    openebs.io/component-name: ndm
    openebs.io/version: 1.3.0
spec:
  selector:
    matchLabels:
      name: openebs-ndm
      openebs.io/component-name: ndm
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: openebs-ndm
        openebs.io/component-name: ndm
        openebs.io/version: 1.3.0
    spec:
      nodeSelector:
        "openebs.io/nodegroup": "storage-node"
      serviceaccountName: openebs-maya-operator
      hostNetwork: true
      containers:
      - name: node-disk-manager
        image: quay.io/openebs/node-disk-manager-amd64:v0.4.3
        imagePullPolicy: Always
        securityContext:
          privileged: true
        volumeMounts:
        - name: config
          mountPath: /host/node-disk-manager.config
          subPath: node-disk-manager.config
          readOnly: true
        - name: udev
          mountPath: /run/udev
        - name: procmount
          mountPath: /host/proc
          readOnly: true
        - name: sparsepath
          mountPath: /var/openebs/sparse
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodename
        - name: SPARSE_FILE_DIR
          value: "/var/openebs/sparse"
        - name: SPARSE_FILE_SIZE
          value: "10737418240"
        - name: SPARSE_FILE_COUNT
          value: "3"
        livenessProbe:
          exec:
            command:
            - pgrep
            - ".*ndm"
          initialDelaySeconds: 30
          periodSeconds: 60
      volumes:
      - name: config
        configMap:
          name: openebs-ndm-config
      - name: udev
        hostPath:
          path: /run/udev
          type: Directory
      - name: procmount
        hostPath:
          path: /proc
          type: Directory
      - name: sparsepath
        hostPath:
          path: /var/openebs/sparse
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openebs-ndm-operator
  namespace: openebs
  labels:
    name: openebs-ndm-operator
    openebs.io/component-name: ndm-operator
    openebs.io/version: 1.3.0
spec:
  selector:
    matchLabels:
      name: openebs-ndm-operator
      openebs.io/component-name: ndm-operator
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: openebs-ndm-operator
        openebs.io/component-name: ndm-operator
        openebs.io/version: 1.3.0
    spec:
      serviceaccountName: openebs-maya-operator
      containers:
        - name: node-disk-operator
          image: quay.io/openebs/node-disk-operator-amd64:v0.4.3
          imagePullPolicy: Always
          readinessProbe:
            exec:
              command:
                - stat
                - /tmp/operator-sdk-ready
            initialDelaySeconds: 4
            periodSeconds: 10
            failureThreshold: 1
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: pod_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # the service account of the ndm-operator pod
            - name: SERVICE_accOUNT
              valueFrom:
                fieldRef:
                  fieldPath: spec.serviceaccountName
            - name: OPERATOR_NAME
              value: "node-disk-operator"
            - name: CLEANUP_JOB_IMAGE
              value: "quay.io/openebs/linux-utils:3.9"
---
apiVersion: v1
kind: Secret
metadata:
  name: admission-server-certs
  namespace: openebs
  labels:
    app: admission-webhook
    openebs.io/component-name: admission-webhook
type: Opaque
data:
  cert.pem: <...pem...>
  key.pem: <...pem...>
---
apiVersion: v1
kind: Service
metadata:
  name: admission-server-svc
  namespace: openebs
  labels:
    app: admission-webhook
    openebs.io/component-name: admission-webhook-svc
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: admission-webhook
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openebs-admission-server
  namespace: openebs
  labels:
    app: admission-webhook
    openebs.io/component-name: admission-webhook
    openebs.io/version: 1.3.0
spec:
  replicas: 1
  strategy:
    type: Recreate
    rollingUpdate: null
  selector:
    matchLabels:
      app: admission-webhook
  template:
    metadata:
      labels:
        app: admission-webhook
        openebs.io/component-name: admission-webhook
        openebs.io/version: 1.3.0
    spec:
      serviceaccountName: openebs-maya-operator
      containers:
        - name: admission-webhook
          image: quay.io/openebs/admission-server:1.3.0
          imagePullPolicy: IfNotPresent
          args:
            - -tlsCertFile=/etc/webhook/certs/cert.pem
            - -tlsKeyFile=/etc/webhook/certs/key.pem
            - -alsologtostderr
            - -v=2
            - 2>&1
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
      volumes:
        - name: webhook-certs
          secret:
            secretName: admission-server-certs
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-cfg
  labels:
    app: admission-webhook
    openebs.io/component-name: admission-webhook
webhooks:
  # failurePolicy Fail means that an error calling the webhook causes the admission to fail.
  - name: admission-webhook.openebs.io
    failurePolicy: Ignore
    clientConfig:
      service:
        name: admission-server-svc
        namespace: openebs
        path: "/validate"
      caBundle: <...ca..>
    rules:
      - operations: [ "CREATE","DELETE" ]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["persistentvolumeclaims"]
      - operations: [ "CREATE","UPDATE" ]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["cstorpoolclusters"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: openebs-localpv-provisioner
  namespace: openebs
  labels:
    name: openebs-localpv-provisioner
    openebs.io/component-name: openebs-localpv-provisioner
    openebs.io/version: 1.3.0
spec:
  selector:
    matchLabels:
      name: openebs-localpv-provisioner
      openebs.io/component-name: openebs-localpv-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: openebs-localpv-provisioner
        openebs.io/component-name: openebs-localpv-provisioner
        openebs.io/version: 1.3.0
    spec:
      serviceaccountName: openebs-maya-operator
      containers:
      - name: openebs-provisioner-hostpath
        imagePullPolicy: Always
        image: quay.io/openebs/provisioner-localpv:1.3.0
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodename
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: OPENEBS_IO_ENABLE_ANALYTICS
          value: "true"
        - name: OPENEBS_IO_INSTALLER_TYPE
          value: "openebs-operator"
        - name: OPENEBS_IO_HELPER_IMAGE
          value: "quay.io/openebs/openebs-tools:3.8"
        livenessProbe:
          exec:
            command:
            - pgrep
            - ".*localpv"
          initialDelaySeconds: 30
          periodSeconds: 60

我在Kubernetes上：

Client Version: version.Info{Major:"1",Minor:"16",GitVersion:"v1.16.2",GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b",GitTreeState:"clean",BuildDate:"2019-10-15T19:18:23Z",GoVersion:"go1.12.10",Compiler:"gc",Platform:"linux/amd64"}
Server Version: version.Info{Major:"1",BuildDate:"2019-10-15T19:09:08Z",Platform:"linux/amd64"}

我如何调查为什么要花这么长时间？ 我还能带些什么？

wxj343302733 回答：OpenEBS cstore卷在删除和覆盖时非常慢

通过Slack向Peter讲话。以下更改有助于提高性能：

  • 增加iSCSI Initiator的队列大小
  • 将三个节点中的每个节点的cpu从2增加到6
  • 创建我自己的sc以使用ext3而不是默认的ext4。

通过上述更改，后续的写入速度约为140-150 MB / s。

kubernetesopenebs
本文链接：https://www.f2er.com/3124627.html

大家都在问