跳到主要内容

· 阅读需 14 分钟
Vincent Lin

KubeEdge 社区已于 2022 年 7 月份完成整个 KubeEdge 项目的第三方安全审计,已发布云原生边缘计算安全威胁分析和防护白皮书,并根据安全威胁模型和安全审计的建议,对 KubeEdge 软件供应链进行持续安全加固。经过社区的不断努力,我们很兴奋地宣布,在 2023 年 1 月 18 日发布的 v1.13.0 版本中,KubeEdge 项目已达到 SLSA L3 等级(包括二进制和容器镜像构件),成为 CNCF 社区首个达到 SLSA L3 等级的项目。

为什么说达到 SLSA L3 等级对 KubeEdge 项目十分重要

软件供应链完整性攻击(对软件包的未经授权的修改)在过去三年中呈上升趋势,KubeEdge 实现 SLSA L3 等级标准后,可以端到端的从源码构建到发布流程进行安全加固,保障用户获取到的二进制或容器镜像产物不被恶意篡改。基于 SLSA 安全框架,可以潜在地加强软件构件的完整性。SLSA 提供端到端的指导原则,可以作为软件的一组防御措施,并防止对组成软件产品的软件包的篡改或任何类型的未经授权的修改。采用 SLSA 框架可以保护项目软件免受常见的供应链攻击。

关于 SLSA

什么是 SLSA(Supply chain Levels for Software Artifacts 软件构件的供应链级别):

Google 提出的用于保证整个软件供应链完整性的框架 SLSA,是一套基于行业共识的安全准则,也是一个安全框架、一份标准和控制清单,用于防止篡改、提高完整性以及保护项目、业务或企业中的软件包和基础设施。它不是一个单一的工具,而是一个逐步采用的大纲,以防止工件被篡改和被篡改的工件被使用,并在更高层次上强化构成供应链的平台。生产商遵循 SLSA 准则使他们的软件更加安全,用户则根据软件包的安全状况来做出决策。

截止目前,SLSA 标准处于 alpha 阶段,相关的定义可能会发生变化。

下图描述了软件供应链中已知的攻击点。更多详细描述,可参考https://slsa.dev/

slsa-dev

SLSA 框架引入了许多新的工具和概念,例如:

  • Artifact(软件制品):由构建流水线生成的任何制品文件,如容器镜像、语言包、编译的二进制文件等;
  • Provenance (来源证据链): 构建的元数据包括构建过程、构建源和依赖关系;
  • Digest (数字摘要):加密哈希函数的结果,该函数生成唯一标识工件的固定大小值,例如容器镜像的 SHA-256 哈希;
  • Attestation (证书):一个加密签名的文件,记录当时生成软件产物的来源;
  • Immutable references(不可变引用-一种标识符):,保证始终指向相同的、不可变的软件制品,如特定的容器图像或语言包;
  • Build integrity(构建完整性):验证构建流水线的输出完整性。

KubeEdge 项目如何达到 SLSA L3

截止目前,SLSA 评估等级共分为 4 个等级 L1~L4,安全性由低到高,每个等级有不同的达标要求,详细的达标要求可参考 SLSA 详细标准(https://slsa.dev/spec/v0.1/requirements)。

在去年 7 月发布的第三方安全审计报告中,KubeEdge 项目在软件供应链 SLSA Provenance 维度暂未达到 L3 等级,经过 SIG-Security 的持续安全加固,在今年 1 月发布的 v1.13.0 版本中,KubeEdge 项目在所有的 SLSA 维度中均达到 L3 等级。以下表格展示了 KubeEdge 在SourceBuildProvenanceCommon中的达标情况(Y 表示 KubeEdge 已达标,空格表示 SLSA 在该等级下未要求)。

SLSA 评估表格及达标情况

RequirementL1L2L3L4
Source
Version controlledYYY
Verified historyYY
Retained indefinitelyYY
Two-person reviewedY
Build
Scripted buildYYYY
Build ServiceYYY
Build as codeYY
Ephemeral environmentYY
IsolatedYY
ParameterlessY
HermeticY
Provenance
AvailableYYYTODO
AuthenticatedYYTODO
Service generatedYYTODO
Non-falsifiableYTODO
Dependencies completeTODO
Common
SecurityY
AccessY
SuperusersY

本章节将着重介绍 KubeEdge 如何达成 SLSA L3 等级在 Build、Provenance 维度的要求。Build/Provenance Requirements 及 KubeEdge 相应的解决方案如下。

Build Requirements:

a) 通过脚本构建:所有的构建步骤都是通过脚本自动化执行。

b) 通过构建服务进行构建:所有的构建步骤由构建服务完成,不在开发者本地环境。构建服务如 GitHub Actions、第三方云平台提供的构建服务等。

c) 作为源码构建:构建服务执行的构建定义文件和配置文件来源于版本控制系统中的文本文件,并且是可验证的。

d) 构建环境临时性:构建服务确保构建步骤在临时环境中运行,例如容器或 VM,仅为此构建提供,而不是复用先前的构建 。

e) 构建的隔离性:构建服务确保构建步骤在隔离的环境中运行,不受其他构建实例(无论是先前的还是并发的)的影响。

f) 无用户自定义参数:除了构建入口点和初始源配置之外,构建输出不会受到用户参数的影响。换句话说,构建完全是通过构建脚本定义的,而不是其他。

g) 封闭性:所有可传递的构建步骤、源和依赖项都使用不可变引用预先完全声明,并且构建步骤在没有网络访问的情况下运行。

解决方案:

KubeEdge 项目所有的构建流程均在 GitHub 上由脚本自动化执行,GitHub Actions 作为构建服务(相关的定义文件和配置文件保存在.github/workflows 目录下),可保障构建过程的可回溯、可验证以及构建环境的临时性、隔离性、构建参数和依赖项不可篡改。

Provenance Requirements:

a) 可用性:Provenance 通过用户可接受的格式提供。应该满足 SLSA Provenance 格式,但如果生产商和用户都同意,并且满足所有其他要求,可以使用另一种格式。

b) 可验证:Provenance 的真实性和完整性可以由用户验证。这应该通过来自私钥的数字签名来实现,只有生成 Provenance 的服务才能访问私钥。

c) 通过构建服务生成:Provenance 中的数据必须从构建服务中获得。

d) 不可伪造:构建服务的用户不能伪造 Provenance。

e) 第三方依赖的完整性:Provenance 记录运行构建步骤时可用的所有构建依赖项。包括构建的机器、VM 或容器的初始状态。

解决方案:

在 KubeEdge 版本发布的产物中,包括二进制文件和容器镜像 2 种格式,通过集成 SLSA 官方的 GitHub 构建项目 slsa-github-generator 来实现 SLSA L3 等级。

在 KubeEdge 版本发布的流程(.github/workflows/release.yml)中,集成了 slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml 和 slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml,可保障构建和发布产物(包括二进制文件和容器镜像)的流程满足 SLSA L3 等级的要求。

更多关于 slsa-github-generator 的详细说明请见https://github.com/slsa-framework/slsa-github-generator

关于 Provenance

Provenance 是构建的元数据包括构建过程、构建源和依赖关系,是软件构建和发布执行流程的一种证明,并且是可以被验证的,包括构建的源码仓库、代码分支、配置文件等信息。在 SLSA L3 级别,Provenance 内容是真实的、防篡改的,并且不会被项目维护者更改。二进制发布产物的 Provenance 文件随发布软件包一起发布,名称为 multiple.intoto.jsonl,容器镜像的 Provenance 文件随镜像文件一起上传到 KubeEdge dockerhub 公开仓库中。具体的 Provenance 格式说明,请参考https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/generic/README.md#provenance-format。

如何校验 KubeEdge 发布产物是否满足 SLSA L3 等级

详细步骤描述请见https://github.com/kubeedge/kubeedge/pull/4285

校验示例如下:

$ COSIGN_EXPERIMENTAL=1 cosign verify-attestation --type slsaprovenance --policy policy.cue kubeedge/cloudcore:v1.13.0
{
"_type": "https://in-toto.io/Statement/v0.1",
"predicateType": "https://slsa.dev/provenance/v0.2",
"subject": [{
"name": "index.docker.io/kubeedge/cloudcore",
"digest": {
"sha256": "825642e63ab5b924e2fa0661cd14d544d0be151c4bdba6f3f42796c977fbe211"
}
}
],
"predicate": {
"builder": {
"id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.4.0"
},
"buildType": "https://github.com/slsa-framework/slsa-github-generator/container@v1",
"invocation": {
"configSource": {
"uri": "git+https://github.com/kubeedge/kubeedge@refs/tags/v1.13.0",
"digest": {
"sha1": "ee357a0d5341241143e88d45df99fde865c987de"
},
"entryPoint": ".github/workflows/release.yml"
},
...

SLSA GitHub generator 签名和验证原理

使用 OpenID Connect (OIDC) 向外部服务 (Sigstore) 证明工作流的身份。OpenID Connect (OIDC) 是身份提供商在网络上使用的标准,用于为第三方证明用户的身份。 GitHub 现在在其工作流程中支持 OIDC。每次运行工作流程时,运行者都可以从 GitHub 的 OIDC 提供商处创建一个唯一的 JWT 令牌。令牌包含工作流身份的可验证信息,包括调用者存储库、提交哈希、触发器以及当前(可重用)工作流路径和引用。

使用 OIDC,工作流向 Sigstore 的 Fulcio 根证书颁发机构证明其身份,后者充当外部验证服务。 Fulcio 签署了一份短期证书,证明运行器中生成的临时签名密钥并将其与工作负载身份相关联。签署出处的记录保存在 Sigstore 的透明日志 Rekor 中。用户可以使用签名证书作为信任方来验证来源是否经过身份验证且不可伪造;它必须是在受信任的构建器中创建的。流程图如下所示。

sigstore

值得一提的是,SLSA GitHub generator 获得 sigstore 社区 2022 年度徽章 Best User Adopter。

best-user-adopter

总结

SLSA 在 KubeEdge 项目软件供应链安全中发挥着重要作用。基于 sigstore 社区提供的能力,从源码到发布产物,对软件供应链端到端的整个流程进行签名和校验,确保 KubeEdge 软件供应链安全。

相关参考:

https://slsa.dev/

https://www.sigstore.dev/

https://github.com/slsa-framework/slsa-github-generator

https://slsa.dev/blog/2022/08/slsa-github-workflows-generic-ga

https://security.googleblog.com/2022/04/improving-software-supply-chain.html

https://blog.sigstore.dev/sigstore-november-roundup-8a852cec10fc/

· 阅读需 13 分钟
Wack Xu

Abstract

The population of KubeEdge brings in community interests in the scalability and scale of KubeEdge. Now, Kubernetes clusters powered by KubeEdge, as fully tested, can stably support 100,000 concurrent edge nodes and manage more than one million pods. This report introduces the metrics used in the test, the test procedure, and the method to connect to an ocean of edge nodes.

Background

Fast growing technologies, such as 5G networks, industrial Internet, and AI, are giving edge computing an important role in driving digital transformation. Cities, transportation, healthcare, manufacturing, and many other fields are becoming smart thanks to edge computing. According to Gartner, by 2023, the number of intelligent edge devices may be more than 20 times that of traditional IT devices. By 2028, the embedding of sensors, storage, computing, and advanced AI functions in edge devices will grow steadily. IoT devices are of various types and in large quantities. The increasing connected IoT devices are challenging management and O&M.

At the same time, users in the KubeEdge community are expecting large-scale edge deployment. There are already some successful use cases for KubeEdge. In unmanned toll stations across China, there are nearly 100,000 edge nodes and more than 500,000 edge applications in this project, and the numbers keep growing. Another case is a vehicle-cloud collaboration platform, the industry-first cloud-edge-device system. It enables fast software upgrade and iteration for software-defined vehicles. On this platform, each vehicle is connected as an edge node, and the number of edge nodes will reach millions.

Introduction to KubeEdge

KubeEdge is the industry's first cloud native edge computing framework designed for edge-cloud collaboration. Complementing Kubernetes for container orchestration and scheduling, KubeEdge allows applications, resources, data, and devices to collaborate between edges and the cloud. Devices, edges, and the cloud are now fully connected in edge computing.

In the KubeEdge architecture, the cloud is a unified control plane, which includes native Kubernetes management components and KubeEdge-developed CloudCore components. It listens to cloud resource changes and provides reliable, efficient cloud-edge messaging. At the edge side lie the EdgeCore components, including Edged, MetaManager, and EdgeHub. They receive messages from the cloud and manage the lifecycle of containers. The device mapper and event bus are responsible for device access.

kubeedge-arch

Based on the Kubernetes control plane, KubeEdge allows nodes to be deployed more remotely and thereby extends edge-cloud collaboration. Kubernetes supports 5,000 nodes and 150,000 pods, which are far from enough for edge computing in Internet of Everything (IoE). The access of a large number of edge devices demands a scalable, centralized edge computing platform. To help users cost less and manage more in an easier way, KubeEdge, fully compatible with Kubernetes, optimizes the cloud-edge messaging and provides access support for mass edge nodes.

SLIs/SLOs

Scalability and performance are important features of Kubernetes clusters. Before performing the large-scale performance test, we need to define the measurement metrics. The Kubernetes community defines the following SLIs (Service Level Indicators) and SLOs (Service Level Objectives) to measure the cluster service quality.

  1. API Call Latency
StatusSLISLO
OfficialLatency of mutating API calls for single objects for every (resource, verb) pair, measured as 99th percentile over last 5 minutesIn default Kubernetes installation, for every (resource, verb) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day <= 1s
OfficialLatency of non-streaming read-only API calls for every (resource, scope) pair, measured as 99th percentile over last 5 minutesIn default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day: (a) <= 1s if scope=resource (b) <= 30s5 otherwise (if scope=namespace or scope=cluster)
  1. Pod Startup Latency
StatusSLISLO
OfficialStartup latency of schedulable stateless pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutesIn default Kubernetes installation, 99th percentile per cluster-day <= 5s
WIPStartup latency of schedulable stateful pods, excluding time to pull images, run init containers, provision volumes (in delayed binding mode) and unmount/detach volumes (from previous pod if needed), measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutesTBD

The community also defines indicators such as in-cluster network programming latency (latency for Service updates or changes in ready pods to be reflected to iptables/IPVS rules), in-cluster network latency, DNS programming latency (latency for Service updates or changes in ready pods to be reflected to the DNS server), and DNS latency. These indicators have not yet been quantified. This test was conducted to satisfy all SLIs/SLOs in the official state.

Kubernetes Scalability Dimensions and Thresholds

Kubernetes scalability does not just mean the number of nodes (Scalability != #Nodes). Other dimensions include the number of namespaces, pods, Services, secrets, and ConfigMaps. Configurations that Kubernetes supports create the Scalability Envelope (which keeps evolving):

k8s-scalability

Obviously, it is impossible for a Kubernetes cluster to expand resource objects without limitation while satisfying SLIs/SLOs. Therefore, the industry defines the upper limits of Kubernetes resource objects.

1. Pods/node 30
2. Backends <= 50k & Services <= 10k & Backends/service <= 250
3. Pod churn 20/s
4. Secret & configmap/node 30
5. Namespaces <= 10k & Pods <= 150k & Pods/namespace <= 3k
6. ​ …..

Dimensions are sometimes not independent. As you move farther along one dimension, your cross-section wrt other dimensions gets smaller. For example, if 5000 nodes are expanded to 10,000 nodes, the specifications of other dimensions will be affected. A heavy workload is required if all scenarios are tested. In this test, we focus on the typical scenarios. We manage to host 100k edge nodes and 1000k pods in a single cluster while satisfying the SLIs/SLOs.

Test Tools

ClusterLoader2

ClusterLoader2 is an open source Kubernetes cluster performance test tool. It can test the Kubernetes SLIs/SLOs to check whether the cluster meets the service quality standards. It also visualizes data for locating cluster problems and optimizing cluster performance. After the test, users get a performance report with detailed test results.

Clusterloader2 performance metrics:

  • APIResponsivenessPrometheusSimple
  • APIResponsivenessPrometheus
  • CPUProfile
  • EtcdMetrics
  • MemoryProfile
  • MetricsForE2E
  • PodStartupLatency
  • ResourceUsageSummary
  • SchedulingMetrics
  • SchedulingThroughput
  • WaitForControlledPodsRunning
  • WaitForRunningPods

Edgemark

Edgemark is a performance test tool similar to Kubemark. It simulates deploying KubeEdge edge nodes in the KubeEdge cluster scalability test to build ultra-large Kubernetes clusters, powered by KubeEdge, with limited resources. The objective is to expose the cluster control plane downsides that occur only in large-scale deployments. The following figure illustrates the Edgemark deployment:

edgemark-deploy

  • K8s master: the master node of the Kubernetes cluster
  • Edgemark master: the master node of the simulated Kubernetes cluster
  • CloudCore: the KubeEdge cloud management component, which is responsible for edge node access
  • hollow pod: a pod started in the actual cluster. It registers with the Edgemark master as a virtual edge node by starting Edgemark in it. The Edgemark master can schedule pods to this virtual edge node.
  • hollow edgeNode: a virtual node in the simulated cluster, registered from a hollow pod

Cluster Deployment Scheme for the Test

deploy

The Kubernetes control plane is deployed with one master node. The etcd, kube-apiserver, kube-scheduler, and kube-controller are deployed as single-instance. The KubeEdge control plane is deployed with five CloudCore instances and connects to the kube-apiserver through the IP address of the master node. Hollow EdgeNodes are exposed by a load balancer and randomly connect to a CloudCore instance based on the round-robin policy of the load balancer.

Test Environment Information

Control Plane OS Version

CentOS 7.9 64bit 3.10.0-1160.15.2.el7.x86_64

Kubernetes Version

Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"

KubeEdge Version

KubeEdge v1.11.0-alpha.0

Master Node Configurations

  • CPU
Architecture:          x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8378A CPU @ 3.00GHz
Stepping: 6
CPU MHz: 2999.998
  • MEMORY
Total online memory:   256G
  • ETCD DISK
Type:   SAS_SSD
Size: 300GB

CloudCore Node Configurations

  • CPU
Architecture:          x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8378A CPU @ 3.00GHz
Stepping: 6
CPU MHz: 2999.998
  • MEMORY
Total online memory:   48G

Component Parameter Configurations

1. kube-apiserver

--max-requests-inflight=2000
--max-mutating-requests-inflight=1000

2. kube-controller-manager

--kube-api-qps=100
--kube-api-burst=100

3. kube-scheduler

--kube-api-qps=200
--kube-api-burst=400

4. CloudCore

apiVersion: cloudcore.config.kubeedge.io/v1alpha1
kind: CloudCore
kubeAPIConfig:
kubeConfig: ""
master: ""
qps: 60000
burst: 80000
modules:
cloudHub:
advertiseAddress:
- xx.xx.xx.xx
nodeLimit: 30000
tlsCAFile: /etc/kubeedge/ca/rootCA.crt
tlsCertFile: /etc/kubeedge/certs/server.crt
tlsPrivateKeyFile: /etc/kubeedge/certs/server.key
unixsocket:
address: unix:///var/lib/kubeedge/kubeedge.sock
enable: false
websocket:
address: 0.0.0.0
enable: true
port: 10000
cloudStream:
enable: false
deviceController:
enable: false
dynamicController:
enable: false
edgeController:
buffer:
configMapEvent: 102400
deletePod: 10240
endpointsEvent: 1
podEvent: 102400
queryConfigMap: 10240
queryEndpoints: 1
queryNode: 10240
queryPersistentVolume: 1
queryPersistentVolumeClaim: 1
querySecret: 10240
queryService: 1
queryVolumeAttachment: 1
ruleEndpointsEvent: 1
rulesEvent: 1
secretEvent: 1
serviceEvent: 10240
updateNode: 15240
updateNodeStatus: 30000
updatePodStatus: 102400
enable: true
load:
deletePodWorkers: 5000
queryConfigMapWorkers: 1000
queryEndpointsWorkers: 1
queryNodeWorkers: 5000
queryPersistentVolumeClaimWorkers: 1
queryPersistentVolumeWorkers: 1
querySecretWorkers: 1000
queryServiceWorkers: 1
queryVolumeAttachmentWorkers: 1
updateNodeStatusWorkers: 10000
updateNodeWorkers: 5000
updatePodStatusWorkers: 20000
ServiceAccountTokenWorkers: 10000
nodeUpdateFrequency: 60
router:
enable: false
syncController:
enable: true

Density Test

Test Execution

Before using ClusterLoader2 to perform the performance test, we defined the test policy using the configuration file. In this test, we used the official Kubernetes density case. The configuration file we used can be obtained here:

https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/testing/density/config.yaml

The following table describes the detailed Kubernetes resource configurations:

Maximum typeMaximum value
Number of Nodes100,000
Number of Pods1,000,000
Number of Pods per node10
Number of Namespaces400
Number of Pods per Namespace2,500

For details about the test method and procedure, see the following links:

https://github.com/kubeedge/kubeedge/tree/master/build/edgemark

https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/GETTING_STARTED.md

Test Results

APIResponsivenessPrometheusSimple

  1. mutating API latency(threshold=1s):

    mutating-api-latency

  2. Read-only API call latency(scope=resource, threshold=1s)

    read-only-api-call-resource

  3. Read-only API call latency(scope=namespace, threshold=5s)

    read-only-api-call-namespace

  4. Read-only API call latency(scope=cluster, threshold=30s)

    read-only-api-call-cluster

PodStartupLatency

metricp50(ms)p90(ms)p99(ms)SLO(ms)
pod_startup1688275140875000
create_to_schedule001000N/A
schedule_to_run100010001000N/A
run_to_watch108716742265N/A
schedule_to_watch165727243070N/A

Note: Theoretically, the latency should always be greater than 0. Because kube-apiserver does not support RFC339NANO, the timestamp precision can only be seconds. Therefore, when the latency is low, some values collected by ClusterLoader2 are 0 due to precision loss.

Conclusion and Analysis

From the preceding test results, the API call latency and pod startup latency meet the SLIs/SLOs defined by the Kubernetes community. Therefore, the KubeEdge-powered Kubernetes clusters can stably support 100,000 concurrent edge nodes and more than one million pods. In production, the network between edge nodes and the cloud is connected according to O&M requirements due to reasons such as network security and partition management. Therefore, the number of edge nodes that can be managed by a single cluster can increase proportionally based on the proportion of offline edge nodes to online ones. In addition, data fragmentation is used on the Kubernetes control plane to store different resources to the corresponding etcd space, which allows for a larger service deployment scale.

KubeEdge's Support for Large-Scale Edge Node Access

1. Efficient Cloud-Edge Messaging

List-watch is a unified mechanism for asynchronous messaging of Kubernetes components. The list operation calls the list API of a resource to obtain full resource data through non-persistent HTTP connections. The watch operation calls the watch API of a resource to monitor resource change events and obtain incremental change data through persistent HTTP connections and block-based transmission encoding. In Kubernetes, in addition to the list-watch of a node, pods allocated to the node, and full service metadata, kubelet must also watch (by default) the running pods mounted with secrets and ConfigMaps as data volumes. The number of list-watch operations could explode with increasing nodes and pods, which heavily burdens kube-apiserver.

KubeEdge uses the two-way multiplexing edge-cloud message channel and supports the WebSocket (default) and QUIC protocols. EdgeCore at the edge initiates a connection request to CloudCore on the cloud. CloudCore list-watches Kubernetes resource changes, and delivers metadata to the edge through this two-way channel. EdgeCore uploads the metadata, such as edge node status and application status, to CloudCore through this channel. CloudCore reports the received metadata to kube-apiserver.

CloudCore aggregates the upstream and downstream data. kube-apiserver processes only several list-watch requests from CloudCore. It can be effectively unburdened and the cluster performance gets improved.

Memory usage when the native Kubernetes kube-apiserver is used under the same node and pod scales:

kube-apiserver-usage

Memory usage when kube-apiserver is used in a KubeEdge-powered Kubernetes cluster:

kubeedge-kube-apiserver-usage

2. Reliable Incremental Cloud-Edge Data Transmission

In the case of complex edge network topology or poor networking quality, cloud-edge communication may be compromised by high network latency, intermittent/frequent disconnection, and other issues. When the network recovers and edge nodes want to reconnect to the cloud, a large number of full list requests will be generated, pressuring kube-apiserver. Large-scale deployments may amplify this challenge to system stability. To solve it, KubeEdge records the version of the metadata successfully sent to the edge. When the cloud-edge network is reconnected, the cloud sends incremental metadata starting from the recorded metadata version.

3. Lightweight Edge + Edge-Cloud Messaging Optimization

EdgeCore removes native kubelet features that are not used in edge deployments, such as in-tree volume and cloud-provider, trims the status information reported by nodes, and optimizes resource usage of edge agent software. EdgeCore can run with a minimum of 70 MB memory on edge devices whose memory is as minimum as 100 MB. The WebSocket channel, edge-cloud message combination, and data trim greatly reduces the communication pressure on the edge and cloud and the access pressure on the control plane. They ensure that the system can work properly even in the case of high latency and jitter.

When 100,000 edge nodes are connected, the number of ELB connections is 100,000.

connect-number

When 100,000 edge nodes and more than 1,000,000 pods are deployed, the inbound rate of the ELB network is about 3 MB/s, and the average uplink bandwidth to each edge node is about 0.25 kbit/s.

network

Next Steps

Targeted tests will be performed on edge devices, edge-cloud messaging, and edge service mesh. In addition, for some edge scenarios, such as large-scale node network disconnection and reconnection, high latency of edge networks, and intermittent disconnection, new SLIs/SLOs need to be introduced to measure the cluster service quality and perform large-scale tests.

· 阅读需 2 分钟
Vincent Lin

As the first cloud-native edge computing community, KubeEdge provides solutions for cloud-edge synergy and has been widely adopted in industries including Transportation, Energy, Internet, CDN, Manufacturing, Smart campus, etc. With the accelerated deployment of KubeEdge in this area based on cloud-edge synergy, the community will improve the security of KubeEdge continuously in cloud-native edge computing scenarios.

The KubeEdge community attaches great importance to security and has set up Sig Security and Security Team to design KubeEdge system security and quickly respond to and handle security vulnerabilities. To conduct a more comprehensive security assessment of the KubeEdge project, the KubeEdge community cooperates with Ada Logics Ltd. and The Open Source Technology Improvement Fund performed a holistic security audit of KubeEdge and output a security auditing report, including the security threat model and security issues related to the KubeEdge project. Thank you to experts Adam Korczynski and David Korczynski of Ada Logics for their professional and comprehensive evaluation of the KubeEdge project, which has important guiding significance for the security protection of the KubeEdge project. Thank you Amir Montazery and Derek Zimmer of OSTIF and Cloud Native Computing Foundation (CNCF) who helped with this engagement.

The discovered security issues have been fixed and patched to the latest three minor release versions (v1.11.1, v1.10.2, v1.9.4) by KubeEdge maintainers according to the kubeedge security policy. Security advisories have been published here.

For more details of the threat model and the mitigations, Please check KubeEdge Threat Model And Security Protection Analysis: https://github.com/kubeedge/community/tree/master/sig-security/sig-security-audit/KubeEdge-threat-model-and-security-protection-analysis.md.

References:

Audit report: https://github.com/kubeedge/community/tree/master/sig-security/sig-security-audit/KubeEdge-security-audit-2022.pdf

OSTIF Blogpost: https://ostif.org/our-audit-of-kubeedge-is-complete-multiple-security-issues-found-and-fixed

CNCF Blogpost:

KubeEdge Threat Model And Security Protection Analysis: https://github.com/kubeedge/community/tree/master/sig-security/sig-security-audit/KubeEdge-threat-model-and-security-protection-analysis.md

· 阅读需 4 分钟
Kevin Wang
Fei Xu

KubeEdge is an open source system extending native containerized application orchestration and device management to hosts at the Edge. It is built upon Kubernetes and provides core infrastructure support for networking, application deployment and metadata synchronization between cloud and edge. It also supports MQTT and allows developers to author custom logic and enable resource constrained device communication at the Edge.

On December 6th, the KubeEdge community is proud to announce the availability of KubeEdge 1.9. This release includes a major upgrade for Custom HTTP Request Routing from Edge to Cloud through ServiceBus for Applications, CloudCore run independently of the Kubernetes Master host and containerized deployment using Helm, EdgeMesh add tls and encryption security, and compiled into rpm package, which includes:

  • Custom HTTP Request Routing from Edge to Cloud through ServiceBus for Applications

  • CloudCore run independently of the Kubernetes Master host

  • EdgeMesh add tls and encryption security

  • Enhance the ease of use of EdgeMesh

  • Support containerized deployment of CloudCore using Helm

  • Support compiled into rpm package and installed on OS such as openEuler using yum package manager

  • 40+ bug fixes and enhancements.

Please refer to CHANGELOG v1.9 for a full list of features in this release

备注

Release details - Release v1.9

备注

How to set up KubeEdge - Setup

Release Highlights

Support Custom HTTP Request Routing from Edge to Cloud through ServiceBus for Applications

A HTTP server is added to ServiceBus, to support custom http request routing from edge to cloud for applications. This simplifies the rest api access with http server on the cloud while client is in the edge.

Refer to the links for more details. (#3254, #3301)

Support CloudCore to run independently of the Kubernetes Master host

CloudCore now supports to run independently of the Kubernetes Master host, iptablesmanager has been added as an independent component, users only need to deploy the iptablesmanager to Kubernetes Master host, which now can add the iptable rules for Cloud-Edge tunnel automatically

Refer to the links for more details. (#3265)

EdgeMesh add tls and encryption security

EdgeMesh's tunnel module adds tls and encryption security capabilities. These features bring more secure protection measures to the user's edgemesh-server component and reduce the risk of edgemesh-server being attacked.

Refer to the links for more details. (EdgeMesh#127)

Enhanced the ease of use of EdgeMesh

EdgeMesh has many improvements in ease of use. Now users can easily deploy EdgeMesh's server and agent components with a single command of helm. At the same time, the restriction on service port naming is removed, and the docker0 dependency is removed, making it easier for users to use EdgeMesh.

Refer to the links for more details. (EdgeMesh#123, EdgeMesh#126, EdgeMesh#136, EdgeMesh#175)

Support containerized deployment of CloudCore using Helm

CloudCore now supports containerized deployment using Helm, which provides better containerized deployment experience.

Refer to the links for more details. (#3265)

Support compiled into rpm package and installed on OS such as openEuler using yum package manager

KubeEdge now supports compiled into rpm package and installed on OS such as openEuler using yum package manager.

Refer to the links for more details. (#3089, #3171)

In addition to the above new features, KubeEdge v1.9 also includes the following enhancements:

  • Rpminstaller: add support for openEuler (#3089)

  • Replaced 'kubeedge/pause' with multi arch image (#3114)

  • Make meta server addr configurable (#3119)

  • Added iptables to Dockerfile and made cloudcore privileged (#3129)

  • Added CustomInterfaceEnabled and CustomInterfaceName for edgecore (#3130)

  • Add experimental feature (#3131)

  • Feat(edge): node ephemeral storage info (#3157)

  • Support envFrom configmap in edge pods (#3176)

  • Update golang to 1.16 (#3190)

  • Metaserver: support shutdown server graceful (#3239)

  • Support labelselector for metaserver (#3262)

Future Outlook

With the release of v1.9, KubeEdge supports custom HTTP request routing from Edge to Cloud through ServiceBus for applications, supports CloudCore running independently of the Kubernetes Master host, supports containerized deployment of CloudCore using Helm, supports tls and encryption security and the ease of use of EdgeMesh. Thanks to Huawei, China Unicom, DaoCloud, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io

· 阅读需 4 分钟
Kevin Wang
Fei Xu

KubeEdge is an open source system extending native containerized application orchestration and device management to hosts at the Edge. It is built upon Kubernetes and provides core infrastructure support for networking, application deployment and metadata synchronization between cloud and edge. It also supports MQTT and allows developers to author custom logic and enable resource constrained device communication at the Edge.

On October 31st, the KubeEdge community is proud to announce the availability of KubeEdge 1.8. This release includes a major upgrade for Active-Active HA Support of CloudCore for Large Scale Cluster, EdgeMesh Architecture Modification, EdgeMesh Cross LAN Communication, and Kubernetes Dependencies Upgrade, which includes:

  • Active-Active HA Support of CloudCore for Large Scale Cluster [Beta]

  • EdgeMesh Architecture Modification

  • EdgeMesh Cross LAN Communication

  • Onvif Device Mapper

  • Kubernetes Dependencies Upgrade

  • 30+ bug fixes and enhancements.

Please refer to CHANGELOG v1.8 for a full list of features in this release

备注

Release details - Release v1.8

备注

How to set up KubeEdge - Setup

Release Highlights

Active-Active HA Support of CloudCore for Large Scale Cluster [Beta]

CloudCore now supports Active-Active HA mode deployment, which provides better scalability support for large scale clusters. Cloud-Edge tunnel can also work with multiple CloudCore instances. CloudCore now can add the iptable rules for Cloud-Edge tunnel automatically.

Refer to the links for more details. (#1560, #2999)

EdgeMesh Architecture Modification

EdgeMesh now has two parts: edgemesh-server and edgemesh-agent. The edgemesh-server requires a public IP address, when users use cross lan communication, it can act as a relay server in the LibP2P mode or assist the agent to establish p2p hole punching. The edgemesh-agent is used to proxy all application traffic of user nodes, acts as an agent for communication between pods at different locations.

Refer to the links for more details. (edgemesh#19)

EdgeMesh Cross LAN Communication

Users can use cross LAN communication feature to implement cross LAN edge to edge application communication and cross LAN edge to cloud application communication.

Refer to the links for more details. (edgemesh#26, edgemesh#37, edgemesh#57)

Onvif Device Mapper

Onvif Device Mapper with Golang implementation is provided, based on new Device Mapper Standard. Users now can use onvif device mapper to manage the ONVIF IP camera.

Refer to the links for more details. (mappers-go#48)

Kubernetes Dependencies Upgrade

Upgrade the vendered kubernetes version to v1.21.4, users now can use the feature of new version on the cloud and on the edge side.

Refer to the links for more details. (#3021, #3034)

In addition to the above new features, KubeEdge v1.8 also includes the following enhancements:

  • Refactor edgesite: import functions and structs instead of copying code (#2893)

  • Avoiding update cm after created a new cm (#2913)

  • Solved the checksum file download problem when ke was installed offline (#2909)

  • cloudcore support configmap dynamic update when the env of container inject from configmap or secret (#2931)

  • Remove edgemesh from edgecore (#2916)

  • keadm: support customsized labels when use join command (#2827)

  • support k8s v1.21.X (#3021)

  • Handling node/*/membership/detail (#3025)

  • sync the response message unconditionally (#3014)

  • support default NVIDIA SMI command (#2680)

Future Outlook

With the release of v1.8, KubeEdge supports Active-Active HA mode deployment, which provides better scalability support for large scale clusters, supports cross LAN communication by EdgeMesh, and supports Onvif Device Mapper. Thanks to Huawei, China Unicom, DaoCloud, Zhejiang University SEL Lab, ARM and other organizations for their contributions, as well as all community contributors for their support!

The community plans to further improve the user experience and the stability of KubeEdge in subsequent versions and create the best “open source” intelligent edge computing platform for everyone to freely use.

For more details regarding KubeEdge, please follow and join us here:

https://kubeedge.io