Source version
Kubernetes v1.5.0
brief introduction
There are various resources in k8s, such as Pod, Service, RC, namespaces and so on. In fact, the user operates on a large number of resources. But these resources are not messy, they are organized in the way of Group Version. Each resource belongs to a group, and there are versions of resources, such as v1, v1beta1, etc.
API groups currently in use by k8s:
"core" group: Its REST path is api/v1
"extensions" group: Its REST path is / apis/extensions/v1beta1
"autoscaling", "abac" ...
The format of the server's URL: / prefix/group/version /... (e.g. / apis/extensions/v1beta1)
Important Structures
API RoupVersion: The organization of API resources, which includes Storage, Group Version, Mapper, Serializer, Convertor and other members. Storage is the interface of etcd, which is a map type, and each resource will establish a connection with etcd; Group Version indicates which group and version the APIGroup Version belongs to; Serializer is used for serialization and deserialization; Convertor provides interfaces for transformation of different versions; Mapper implements the RESTMapper interface.
type APIGroupVersion struct { // The key exists in the url of the object, value is a rest.Storage for docking etcd storage Storage map[string]rest.Storage // The prefix of the group, for example, the Root of the core group is'/ api' Root string // Contains string s like'api/v1'to identify the instance GroupVersion unversioned.GroupVersion // OptionsExternalVersion controls the Kubernetes APIVersion used for common objects in the apiserver // schema like api.Status, api.DeleteOptions, and api.ListOptions. Other implementors may // define a version "v1beta1" but want to use the Kubernetes "v1" internal objects. If // empty, defaults to GroupVersion. OptionsExternalVersion *unversioned.GroupVersion // Key members Mapper meta.RESTMapper // Object serialization and deserializer Serializer runtime.NegotiatedSerializer ParameterCodec runtime.ParameterCodec // The following four are assigned to Scheme structures Typer runtime.ObjectTyper Creater runtime.ObjectCreater // To convert any version of the api to each other, you need to register the conversion function beforehand Convertor runtime.ObjectConvertor Copier runtime.ObjectCopier Linker runtime.SelfLinker // Used for access permission control Admit admission.Interface Context api.RequestContextMapper MinRequestTimeout time.Duration // SubresourceGroupVersionKind contains the GroupVersionKind overrides for each subresource that is // accessible from this API group version. The GroupVersionKind is that of the external version of // the subresource. The key of this map should be the path of the subresource. The keys here should // match the keys in the Storage map above for subresources. SubresourceGroupVersionKind map[string]unversioned.GroupVersionKind // ResourceLister is an interface that knows how to list resources // for this API Group. ResourceLister APIResourceLister }
APIGroup Version is a new APIGroup Version () interface in pkg/genericapiserver/genericapiserver.go. Several other structures are used to create APIGroup Version: APIGroup Info, Scheme, GroupMeta. The following is an introduction:
APIGroupInfo:
type APIGroupInfo struct { // Meta-information of the Group GroupMeta apimachinery.GroupMeta // All Storage in Different Versions VersionedResourcesStorageMap map[string]map[string]rest.Storage // OptionsExternalVersion controls the APIVersion used for common objects in the // schema like api.Status, api.DeleteOptions, and api.ListOptions. Other implementors may // define a version "v1beta1" but want to use the Kubernetes "v1" internal objects. // If nil, defaults to groupMeta.GroupVersion. // TODO: Remove this when https://github.com/kubernetes/kubernetes/issues/19018 is fixed. OptionsExternalVersion *unversioned.GroupVersion // In the case of core group, the corresponding is api.Scheme. Scheme *runtime.Scheme // NegotiatedSerializer controls how this group encodes and decodes data NegotiatedSerializer runtime.NegotiatedSerializer // ParameterCodec performs conversions for query parameters passed to API calls ParameterCodec runtime.ParameterCodec // For all resources information, key is the path of resource // For example: key is "replication Controllers / scale", Group Version Kind: autoscaling, v1, Scale SubresourceGroupVersionKind map[string]unversioned.GroupVersionKind }
Scheme: Used for serialization, deserialization, version conversion between API resources. There are several map s in Scheme. The previous structure stores unversioned.GroupVersionKind, unversioned.GroupVersion, which essentially represent the string identifier of the resource. Scheme stores the structure of the specific API resource corresponding to the tag, namely relect.Type.
type Scheme struct { // versionMap allows one to figure out the go type of an object with // the given version and name. gvkToType map[unversioned.GroupVersionKind]reflect.Type // typeToGroupVersion allows one to find metadata for a given go object. // The reflect.Type we index by should *not* be a pointer. typeToGVK map[reflect.Type][]unversioned.GroupVersionKind // unversionedTypes are transformed without conversion in ConvertToVersion. unversionedTypes map[reflect.Type]unversioned.GroupVersionKind // unversionedKinds are the names of kinds that can be created in the context of any group // or version // TODO: resolve the status of unversioned types. unversionedKinds map[string]reflect.Type // Map from version and resource to the corresponding func to convert // resource field labels in that version to internal version. fieldLabelConversionFuncs map[string]map[string]FieldLabelConversionFunc // defaulterFuncs is an array of interfaces to be called with an object to provide defaulting // the provided object must be a pointer. defaulterFuncs map[reflect.Type]func(interface{}) // converter stores all registered conversion functions. It also has // default coverting behavior. converter *conversion.Converter // cloner stores all registered copy functions. It also has default // deep copy behavior. cloner *conversion.Cloner }
GroupMeta: It mainly includes the meta-information of Group, the member RESTMapper in which, like APIGroup Version, the REST Mapper of APIGroup Version is directly taken from the REST Mapper of Group Meta. A Group may contain multiple versions and be stored in Group Version, which is the default version stored in etcd.
type GroupMeta struct { // Default version GroupVersion unversioned.GroupVersion // There may be multiple versions in this Group, and this field contains all versions. GroupVersions []unversioned.GroupVersion // Used for encoding and decoding Codec runtime.Codec // SelfLinker can set or get the SelfLink field of all API types. // TODO: when versioning changes, make this part of each API definition. // TODO(lavalamp): Combine SelfLinker & ResourceVersioner interfaces, force all uses // to go through the InterfacesFor method below. SelfLinker runtime.SelfLinker // For type, conversion between objects RESTMapper meta.RESTMapper // InterfacesFor returns the default Codec and ResourceVersioner for a given version // string, or an error if the version is not known. // TODO: make this stop being a func pointer and always use the default // function provided below once every place that populates this field has been changed. InterfacesFor func(version unversioned.GroupVersion) (*meta.VersionInterfaces, error) // InterfacesByVersion stores the per-version interfaces. InterfacesByVersion map[unversioned.GroupVersion]*meta.VersionInterfaces }
RESTMapper: Used to manage information for all objects. For external acquisition, RESTMapper can be obtained directly through version and group, and then corresponding information can be obtained through kind type. The RESTMapper actually implements a Default RESTMapper structure.
type DefaultRESTMapper struct { defaultGroupVersions []unversioned.GroupVersion resourceToKind map[unversioned.GroupVersionResource]unversioned.GroupVersionKind kindToPluralResource map[unversioned.GroupVersionKind]unversioned.GroupVersionResource kindToScope map[unversioned.GroupVersionKind]RESTScope singularToPlural map[unversioned.GroupVersionResource]unversioned.GroupVersionResource pluralToSingular map[unversioned.GroupVersionResource]unversioned.GroupVersionResource interfacesFunc VersionInterfacesFunc // aliasToResource is used for mapping aliases to resources aliasToResource map[string][]string }
APIRegistration Manager: This structure mainly provides the concept of registered, aggregates all registered, activated, third-party GroupVersions, and also includes the GroupMeta (metadata) of each GroupVersion.
type APIRegistrationManager struct { // So GroupVersions already registered registeredVersions map[unversioned.GroupVersion]struct{} // GroupVersions registered by third parties, which are dynamically registered with apiServer thirdPartyGroupVersions []unversioned.GroupVersion // All GroupVersion s that have enabled can be added through enable Versions (). // Only enable d can you use the corresponding GroupVersion enabledVersions map[unversioned.GroupVersion]struct{} // GroupMeta for all groups groupMetaMap map[string]*apimachinery.GroupMeta // Related to the environment variable'KUBE_API_VERSIONS' envRequestedVersions []unversioned.GroupVersion }
APIRegistration Manager initialization
Path of the structure: pkg/apimachinery/registered/registered.go
In this file we can see that a DefaultAPIRegistration Manager object is initialized:
var ( DefaultAPIRegistrationManager = NewOrDie(os.Getenv("KUBE_API_VERSIONS")) )
Enter the NewOrDie() interface and see:
func NewOrDie(kubeAPIVersions string) *APIRegistrationManager { m, err := NewAPIRegistrationManager(kubeAPIVersions) if err != nil { glog.Fatalf("Could not construct version manager: %v (KUBE_API_VERSIONS=%q)", err, kubeAPIVersions) } return m } func NewAPIRegistrationManager(kubeAPIVersions string) (*APIRegistrationManager, error) { m := &APIRegistrationManager{ registeredVersions: map[unversioned.GroupVersion]struct{}{}, thirdPartyGroupVersions: []unversioned.GroupVersion{}, enabledVersions: map[unversioned.GroupVersion]struct{}{}, groupMetaMap: map[string]*apimachinery.GroupMeta{}, envRequestedVersions: []unversioned.GroupVersion{}, } // If the environment variable KUBE_API_VERSIONS is set, traverse if len(kubeAPIVersions) != 0 { // Separation by commas for _, version := range strings.Split(kubeAPIVersions, ",") { // Parse version and convert it to GroupVersion format // Usually the version here is in group/version format, such as'/ api/v1' gv, err := unversioned.ParseGroupVersion(version) if err != nil { return nil, fmt.Errorf("invalid api version: %s in KUBE_API_VERSIONS: %s.", version, kubeAPIVersions) } // Then put the gv into envRequested Versions m.envRequestedVersions = append(m.envRequestedVersions, gv) } } // Otherwise, return an empty APIRegistration Manager return m, nil }
Looking at the environment we are using, we did not configure KUBE_API_VERSIONS, which returned an empty structure and provided many methods.
var ( ValidateEnvRequestedVersions = DefaultAPIRegistrationManager.ValidateEnvRequestedVersions AllPreferredGroupVersions = DefaultAPIRegistrationManager.AllPreferredGroupVersions RESTMapper = DefaultAPIRegistrationManager.RESTMapper GroupOrDie = DefaultAPIRegistrationManager.GroupOrDie AddThirdPartyAPIGroupVersions = DefaultAPIRegistrationManager.AddThirdPartyAPIGroupVersions IsThirdPartyAPIGroupVersion = DefaultAPIRegistrationManager.IsThirdPartyAPIGroupVersion RegisteredGroupVersions = DefaultAPIRegistrationManager.RegisteredGroupVersions IsRegisteredVersion = DefaultAPIRegistrationManager.IsRegisteredVersion IsRegistered = DefaultAPIRegistrationManager.IsRegistered Group = DefaultAPIRegistrationManager.Group EnabledVersionsForGroup = DefaultAPIRegistrationManager.EnabledVersionsForGroup EnabledVersions = DefaultAPIRegistrationManager.EnabledVersions IsEnabledVersion = DefaultAPIRegistrationManager.IsEnabledVersion IsAllowedVersion = DefaultAPIRegistrationManager.IsAllowedVersion EnableVersions = DefaultAPIRegistrationManager.EnableVersions RegisterGroup = DefaultAPIRegistrationManager.RegisterGroup RegisterVersions = DefaultAPIRegistrationManager.RegisterVersions InterfacesFor = DefaultAPIRegistrationManager.InterfacesFor )
When analyzing the startup process of apiServer, you will find that many variables above are used to initialize the ServerRunOptions object, such as:
Path: pkg/genericapiserver/options/server_run_options.go
func NewServerRunOptions() *ServerRunOptions { return &ServerRunOptions{ AdmissionControl: "AlwaysAdmit", . . . // The AllPreferred Group Versions interface is used here. DefaultStorageVersions: registered.AllPreferredGroupVersions(), . . . StorageVersions: registered.AllPreferredGroupVersions(), } }
The registered. AllPreferred GroupVersions () interface is used above. By the way, the interface is implemented in detail.
func (m *APIRegistrationManager) AllPreferredGroupVersions() string { // If groupMeta is not registered, this is == 0 // But it's impossible not to register. As for where to register, you have to look at the GroupMeta initialization described later. if len(m.groupMetaMap) == 0 { return "" } var defaults []string for _, groupMeta := range m.groupMetaMap { defaults = append(defaults, groupMeta.GroupVersion.String()) } sort.Strings(defaults) return strings.Join(defaults, ",") }
The interface is relatively simple, that is to extract all groupMeta from m.groupMetaMap, and then splice it into a string of "group1/version 1, group2/version 2,..." by comma.
Think about it here. Since there is a list, there must be groupMeta. And let's look at the initialization of APIRegistration Manager. Without setting the KUBE_API_VERSIONS environment variable, there is no groupMeta at all.
Since it's impossible to do without groupMeta, register & enable must be done elsewhere. We can start with the RegisterGroup method provided by APIRegistration Manager:
func (m *APIRegistrationManager) RegisterGroup(groupMeta apimachinery.GroupMeta) error { groupName := groupMeta.GroupVersion.Group if _, found := m.groupMetaMap[groupName]; found { return fmt.Errorf("group %v is already registered", m.groupMetaMap) } m.groupMetaMap[groupName] = &groupMeta return nil }
The input to the RegisterGroup interface is GroupMeta, so we have to continue to look at the initialization of the structure.
GroupMeta initialization
At the present stage of k8s, API is divided into 13 groups: Core, apps, authentication, authorization, autoscaling, batch, certificates, component config, extensions, image policy, policy, rbac, storage. Core's Group Name is empty. It contains APIs that are the core APIs, such as Pod, Service, etc. Its code is under pkg/api, and the other 12 groups are under pkg/apis. Each directory has an install directory with an install.go file, which is then initialized through init(). These programs are import ed through the following files:
Path: pkg/master/import_known_versions.go
package master // These imports are the API groups the API server will support. import ( "fmt" _ "k8s.io/kubernetes/pkg/api/install" "k8s.io/kubernetes/pkg/apimachinery/registered" _ "k8s.io/kubernetes/pkg/apis/apps/install" _ "k8s.io/kubernetes/pkg/apis/authentication/install" _ "k8s.io/kubernetes/pkg/apis/authorization/install" _ "k8s.io/kubernetes/pkg/apis/autoscaling/install" _ "k8s.io/kubernetes/pkg/apis/batch/install" _ "k8s.io/kubernetes/pkg/apis/certificates/install" _ "k8s.io/kubernetes/pkg/apis/componentconfig/install" _ "k8s.io/kubernetes/pkg/apis/extensions/install" _ "k8s.io/kubernetes/pkg/apis/imagepolicy/install" _ "k8s.io/kubernetes/pkg/apis/policy/install" _ "k8s.io/kubernetes/pkg/apis/rbac/install" _ "k8s.io/kubernetes/pkg/apis/storage/install" )
A total of 13 groups were import ed. Among them, "k8s.io/kubernetes/pkg/api/install" is the Core Group. Let's take it as an example and look at the corresponding install.go file.
Path: pkg/api/install/install.go
var availableVersions = []unversioned.GroupVersion{v1.SchemeGroupVersion} func init() { // To register Versions is to deposit them in APIRegistration Manager. registered Versions. registered.RegisterVersions(availableVersions) externalVersions := []unversioned.GroupVersion{} for _, v := range availableVersions { // Determine whether it has been registered and add it to a slice if registered.IsAllowedVersion(v) { externalVersions = append(externalVersions, v) } } if len(externalVersions) == 0 { glog.V(4).Infof("No version is registered for group %v", api.GroupName) return } // enable is actually stored in APIRegistration Manager. enabled Versions if err := registered.EnableVersions(externalVersions...); err != nil { glog.V(4).Infof("%v", err) return } // This interface is key, and it is introduced separately. if err := enableVersions(externalVersions); err != nil { glog.V(4).Infof("%v", err) return } }
Firstly, a slice available Versions is defined, in which there is only one element v1. SchemeGroup Version:
const GroupName = "" var SchemeGroupVersion = unversioned.GroupVersion{Group: GroupName, Version: "v1"}
According to the definition of this element, we can see that available Versions defines a GroupName as empty, and Version as GroupVersion of'v1'. The Group Version is then put into the registered Versions and enabled Versions of APIRegistration Manager.
Several interface implementations of registered are relatively simple and not introduced, but enableVersions() execution is a major issue, and we continue to go deeper:
func enableVersions(externalVersions []unversioned.GroupVersion) error { // Literally: Add all Versions to Scheme // Initialization of Scheme will be introduced later. // The deeper you look at the concepts involved, the more important the interface is and the more patiently you need to mine it. addVersionsToScheme(externalVersions...) // Use a GroupVersion as the default,'/api/v1' preferredExternalVersion := externalVersions[0] // Here it is! GroupMeta was initialized. That's the key to this section. groupMeta := apimachinery.GroupMeta{ GroupVersion: preferredExternalVersion, GroupVersions: externalVersions, // RESTMapper is also the key, which is described in a separate section below. RESTMapper: newRESTMapper(externalVersions), SelfLinker: runtime.SelfLinker(accessor), InterfacesFor: interfacesFor, } // The first two are registers and enable d versions. Here is the register of Group. // The interface actually takes the groupName of the first GroupVersion as the key and groupMeta as the value. // The group MetaMap of APIRegistration Manager is assigned. if err := registered.RegisterGroup(groupMeta); err != nil { return err } return nil }
At this point, combined with the previous APIRegistration Manager initialization, we can see that there should be several groups of groupMetaMap s. The registered. AllPreferred GroupVersions () interface called in the Server RunOptions object initialization can return several DefaultStorage Versions, at least certainly'/ api/v1'. As for the other groupMeta, we need to look at the other install.go, so we won't start talking about the similarities and differences one by one.
groupMeta's initialization is over, but here's another key Scheme, so move on to the next section.
Scheme initialization
When introducing the enableVersions() function in the previous section, the first line is to call addVersionsToScheme(externalVersions...) and add GroupVersions to Scheme. Let's look at the interface:
func addVersionsToScheme(externalVersions ...unversioned.GroupVersion) { // add the internal version to Scheme if err := api.AddToScheme(api.Scheme); err != nil { // Programmer error, detect immediately panic(err) } // add the enabled external versions to Scheme for _, v := range externalVersions { if !registered.IsEnabledVersion(v) { glog.Errorf("Version %s is not enabled, so it will not be added to the Scheme.", v) continue } switch v { case v1.SchemeGroupVersion: if err := v1.AddToScheme(api.Scheme); err != nil { // Programmer error, detect immediately panic(err) } } } }
In the interface, we can see that AddToScheme(api.Scheme) adds GroupVersion to api.Scheme. Let's first put the above interface parsing in place and see how api.Scheme is initialized:
Path: pkg/api/register.go
var Scheme = runtime.NewScheme()
Define Scheme, and then look at NewScheme():___________
Path: pkg/runtime/scheme.go
func NewScheme() *Scheme { // Define empty Scheme s := &Scheme{ gvkToType: map[unversioned.GroupVersionKind]reflect.Type{}, typeToGVK: map[reflect.Type][]unversioned.GroupVersionKind{}, unversionedTypes: map[reflect.Type]unversioned.GroupVersionKind{}, unversionedKinds: map[string]reflect.Type{}, cloner: conversion.NewCloner(), fieldLabelConversionFuncs: map[string]map[string]FieldLabelConversionFunc{}, defaulterFuncs: map[reflect.Type]func(interface{}){}, } // Create converter s for object conversion in different versions s.converter = conversion.NewConverter(s.nameFunc) // Adding some transformation functions s.AddConversionFuncs(DefaultEmbeddedConversions()...) // Enable map[string][]string conversions by default if err := s.AddConversionFuncs(DefaultStringConversions...); err != nil { panic(err) } if err := s.RegisterInputDefaults(&map[string][]string{}, JSONKeyMapper, conversion.AllowDifferentFieldTypeNames|conversion.IgnoreMissingFields); err != nil { panic(err) } if err := s.RegisterInputDefaults(&url.Values{}, JSONKeyMapper, conversion.AllowDifferentFieldTypeNames|conversion.IgnoreMissingFields); err != nil { panic(err) } return s }
An empty Scheme is created above.
Knowing where to create Scheme, let's go back to the addVersionsToScheme() function above.
Actually, it mainly looks at two interfaces: api.AddToScheme() and v1.AddToScheme().
Look at the first one:
var ( SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes, addDefaultingFuncs) AddToScheme = SchemeBuilder.AddToScheme )
Two functions are passed in through the runtime.NewSchemeBuilder() interface, and SchemeBuilder is created:
type SchemeBuilder []func(*Scheme) error func (sb *SchemeBuilder) Register(funcs ...func(*Scheme) error) { for _, f := range funcs { *sb = append(*sb, f) } } func NewSchemeBuilder(funcs ...func(*Scheme) error) SchemeBuilder { var sb SchemeBuilder sb.Register(funcs...) return sb }
According to the above definition and function, SchemeBuilder is an interface slice, which includes two interfaces: addKnownTypes and addDefaulting Funcs.
After SchemeBuilder has been defined, continue to look at AddToScheme:
func (sb *SchemeBuilder) AddToScheme(s *Scheme) error { for _, f := range *sb { if err := f(s); err != nil { return err } } return nil }
This function calls addKnownTypes and addDefaulting Funcs. Let's look at them one by one.
func addKnownTypes(scheme *runtime.Scheme) error { if err := scheme.AddIgnoredConversionType(&unversioned.TypeMeta{}, &unversioned.TypeMeta{}); err != nil { return err } // Add the following objects to Scheme // The Group Name of the Scheme Group Version is empty and the Version is "_internal" // So the interface actually adds k8s built-in version to Scheme, and each group has this step scheme.AddKnownTypes(SchemeGroupVersion, &Pod{}, &PodList{}, &PodStatusResult{}, &PodTemplate{}, &PodTemplateList{}, &ReplicationControllerList{}, &ReplicationController{}, &ServiceList{}, &Service{}, &ServiceProxyOptions{}, &NodeList{}, &Node{}, &NodeProxyOptions{}, &Endpoints{}, &EndpointsList{}, &Binding{}, &Event{}, &EventList{}, &List{}, &LimitRange{}, &LimitRangeList{}, &ResourceQuota{}, &ResourceQuotaList{}, &Namespace{}, &NamespaceList{}, &ServiceAccount{}, &ServiceAccountList{}, &Secret{}, &SecretList{}, &PersistentVolume{}, &PersistentVolumeList{}, &PersistentVolumeClaim{}, &PersistentVolumeClaimList{}, &DeleteOptions{}, &ListOptions{}, &PodAttachOptions{}, &PodLogOptions{}, &PodExecOptions{}, &PodProxyOptions{}, &ComponentStatus{}, &ComponentStatusList{}, &SerializedReference{}, &RangeAllocation{}, &ConfigMap{}, &ConfigMapList{}, ) // In GroupName empty and Version "v1" groupVersion, add these objects to Scheme scheme.AddUnversionedTypes(Unversioned, &unversioned.ExportOptions{}, &unversioned.Status{}, &unversioned.APIVersions{}, &unversioned.APIGroupList{}, &unversioned.APIGroup{}, &unversioned.APIResourceList{}, ) return nil }
View the AddKnownTypes() interface:
func (s *Scheme) AddKnownTypes(gv unversioned.GroupVersion, types ...Object) { if len(gv.Version) == 0 { panic(fmt.Sprintf("version is required on all types: %s %v", gv, types[0])) } for _, obj := range types { t := reflect.TypeOf(obj) if t.Kind() != reflect.Ptr { panic("All types must be pointers to structs.") } t = t.Elem() if t.Kind() != reflect.Struct { panic("All types must be pointers to structs.") } gvk := gv.WithKind(t.Name()) s.gvkToType[gvk] = t s.typeToGVK[t] = append(s.typeToGVK[t], gvk) } }
The interface mainly operates s s s.gvkToType and s.typeToGVK for conversion purposes.
To sum up, internal version is added to Scheme.
Why is there an internal version? In fact, every Group has an internal version. The apiserver operation is also internal version.
For example, if a request to create a Pod comes, the apiserver will first de-serialize the request. User-sent Pod requests are often versioned, such as v1, so they will be de-serialized into a v1.Pod. Apiserver will immediately convert v1.Pod into internal.Pod using convertor, and then do some operations. Finally, it will be stored in etcd. Pod information in etcd is versioned, so a conversion will take place first, and it will be converted to v1.Pod, and then serialized into etcd.
Does this look like a superfluous move? In fact, this is k8s support for multiple versions of api, so that users can create a Pod with a v1beta1, and then put it into etcd is a relatively stable version, such as v1 version.
After the internal version is added, go back to the original addVersionsToScheme() function, and continue to execute the v1.AddToScheme(api.Scheme) function. In fact, the V1 version of the API is added to Scheme, just like the internal version.
Let's look at v1.AddToScheme.
Path: pkg/api/v1/register.go
var ( SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes, addDefaultingFuncs, addConversionFuncs, addFastPathConversionFuncs) AddToScheme = SchemeBuilder.AddToScheme )
Here you can see that v1 has several functions addConversion Funcs, addFastPathConversion Funcs, compared with the internal version.
These functions actually have to traverse the execution of AddToScheme(), which can be looked at in depth. In fact, it adds conversion functions to Scheme, such as converting v1.Pod to internal.Pod and converting internal.Pod to v1.Pod. If there are v1, V2 and V3 at the same time, how will V3 be converted? In fact, it is still a unified conversion to internal, and then to the corresponding version (v1,v2,v3). So internal is equivalent to the bridge of conversion, better support for different versions of api.
At this point, the initialization of Scheme is basically over. The key RESTMapper was also introduced when GroupMeta was initialized, so I'll continue.
RESTMapper initialization
Initialization of this section directly looks at the interface called at GroupMeta initialization, newRESTMapper():__________
Path: pkg/api/install/install.go
func newRESTMapper(externalVersions []unversioned.GroupVersion) meta.RESTMapper { // These are the top-level objects in the API, which can be understood as objects without namespace // Objects are classified into two categories based on whether namespace exists or not: RESTScopeNamespace and RESTScopeRoot rootScoped := sets.NewString( "Node", "Namespace", "PersistentVolume", "ComponentStatus", ) // Need to ignore the following types in Scheme ignoredKinds := sets.NewString( "ListOptions", "DeleteOptions", "Status", "PodLogOptions", "PodExecOptions", "PodAttachOptions", "PodProxyOptions", "NodeProxyOptions", "ServiceProxyOptions", "ThirdPartyResource", "ThirdPartyResourceData", "ThirdPartyResourceList") mapper := api.NewDefaultRESTMapper(externalVersions, interfacesFor, importPrefix, ignoredKinds, rootScoped) return mapper }
In fact, all api resources can be divided into two categories: one is namespace, the other is namespace. For example, Node, Namespace, PersistentVolume, Component Status in this interface do not belong to any namespace. ignoredKinds are the parameters needed for the following interfaces to indicate that these types are ignored when traversing Scheme.
Then call api.NewDefaultRESTMapper(), and importPrefix parameter is "k8s.io/kubernetes/pkg/api".
InterfacesForis an interface.
Path: pkg/api/mapper.go
func NewDefaultRESTMapper(defaultGroupVersions []unversioned.GroupVersion, interfacesFunc meta.VersionInterfacesFunc, importPathPrefix string, ignoredKinds, rootScoped sets.String) *meta.DefaultRESTMapper { // Join Scheme and continue calling the following interface return NewDefaultRESTMapperFromScheme(defaultGroupVersions, interfacesFunc, importPathPrefix, ignoredKinds, rootScoped, Scheme) } func NewDefaultRESTMapperFromScheme(defaultGroupVersions []unversioned.GroupVersion, interfacesFunc meta.VersionInterfacesFunc, importPathPrefix string, ignoredKinds, rootScoped sets.String, scheme *runtime.Scheme) *meta.DefaultRESTMapper { // Initialize a DefaultRESTMapper object mapper := meta.NewDefaultRESTMapper(defaultGroupVersions, interfacesFunc) // Walk through all types from Scheme based on defaultGroupVersions entered, such as "/ api/v1" // Then Add for _, gv := range defaultGroupVersions { for kind, oType := range scheme.KnownTypes(gv) { gvk := gv.WithKind(kind) // Filter out APIs that do not belong to the "k8s.io/kubernetes/pkg/api" path, and ignoredKinds if !strings.Contains(oType.PkgPath(), importPathPrefix) || ignoredKinds.Has(kind) { continue } // Determine whether the kind has a namespace attribute scope := meta.RESTScopeNamespace if rootScoped.Has(kind) { scope = meta.RESTScopeRoot } // Then add the gvk to the corresponding group mapper.Add(gvk, scope) } } return mapper }
Looking at the interface again, first create an empty DefaultRESTMapper, then traverse all kinds of Scheme according to the group Version of'/ api/v1', then call mapper.Add(gvk, scope) to fill the mapper, and finally return to the mapper.
Look at the implementation of mapper.Add():
func (m *DefaultRESTMapper) Add(kind unversioned.GroupVersionKind, scope RESTScope) { // resource is also divided into singular and complex numbers plural, singular := KindToResource(kind) // Singular and Complex Conversion m.singularToPlural[singular] = plural m.pluralToSingular[plural] = singular // Find the corresponding kind according to the resource of singular and complex numbers m.resourceToKind[singular] = kind m.resourceToKind[plural] = kind // Find the corresponding singular and plural resource s according to kind m.kindToPluralResource[kind] = plural // The Conversion from kind to scope m.kindToScope[kind] = scope }
RESTMapper actually contains a transformation relationship, from resource to kind, from kind to resource, and from kind to scope. Resource also has singular and plural numbers.
What's the difference between kind and resource? Both of them are strings. Kind takes its value through Kind = reflector. TypeOf (& Pod {}). Elem (). Name () to get the name of the structure Pod. Resource is valued by plural, singular: = KindToResource (kind). Singular is the conversion of Kind to lowercase letters, while plural is the plural.
Example: Take Pod as an example. Kind is {Group:"", "Version:" "v1", "Kind:" Pod". Then singular is {Group:", "Version:", "v1", "Kind:" pod"}, plural is {Group:", "Version:" v1","Resources:"pods"}.
resource distinguishes between singular and plural numbers in order to obtain Pods information. For example, kubectl get pods or kubectl get pods.
At this point, RESTMapper is basically initialized. From all the initialization above, we can see that it is mainly used to fill Scheme with internal version and external version, and GroupMeta and its members RESTMapper with external version.
What role does GroupMeta play? It is mainly used to initialize APIGroup Version.
API resources are registered as restful api
Previous initializations were intended to pave the way for this step. There is also an API RoupInfo and API Group Version that are not covered. This section will appear.
When the initialization of API resources is completed, these API resources need to be registered as restful APIs to receive user requests.
kube-apiServer uses go-restful framework, which includes three main objects:
Container: A Container contains multiple Web Services
WebService: A Web Service contains multiple route s
Route: A route contains a method(GET, POST, DELETE, etc.), a specific path(URL) and a handler function for the response.
There are two entry functions for API registration: M. Install APIs and m. Install Legacy APIs.
File path: pkg/master/master.go
These two functions are used to register the APIs of'/ api'and'/ apis' respectively. Here we introduce the Install Legacy API first.
These interfaces are called in the config.Complete().New() function:
restOptionsFactory := restOptionsFactory{ deleteCollectionWorkers: c.DeleteCollectionWorkers, enableGarbageCollection: c.GenericConfig.EnableGarbageCollection, storageFactory: c.StorageFactory, } // Determine whether Cache for Watch is enabled // There are different interface implementations with or without cache assignment // restOptionsFactory.storageDecorator: A REST interface(CRUD) decorator for various resources // This interface will be used when calling NewStorage(), and the corresponding CRUD interface and destruction interface will be output. // Refer to NewStorage() in pkg/registry/core/pod/etcd/etcd.go. // In fact, the interface difference between cache and non-cache is that if there is cache, it provides the interface to operate cache; if there is no cache, it provides the interface to operate etcd directly. if c.EnableWatchCache { restOptionsFactory.storageDecorator = registry.StorageWithCacher } else { restOptionsFactory.storageDecorator = generic.UndecoratedStorage } // Determine whether / api/v1 group has been registered and enable d, and then install if c.GenericConfig.APIResourceConfigSource.AnyResourcesForVersionEnabled(apiv1.SchemeGroupVersion) { // This object mainly provides an interface for NewLegacy RESTStorage (). legacyRESTStorageProvider := corerest.LegacyRESTStorageProvider{ StorageFactory: c.StorageFactory, ProxyTransport: c.ProxyTransport, KubeletClientConfig: c.KubeletClientConfig, EventTTL: c.EventTTL, ServiceIPRange: c.ServiceIPRange, ServiceNodePortRange: c.ServiceNodePortRange, LoopbackClientConfig: c.GenericConfig.LoopbackClientConfig, } // Perform API installation of "/api/v1" m.InstallLegacyAPI(c.Config, restOptionsFactory.NewFor, legacyRESTStorageProvider) }
Continue to view the m. Install Legacy API ():
func (m *Master) InstallLegacyAPI(c *Config, restOptionsGetter genericapiserver.RESTOptionsGetter, legacyRESTStorageProvider corerest.LegacyRESTStorageProvider) { // This object has been introduced before. It's more critical and needs to be looked at in depth. // Returns RESTStorage and apiGroupInfo, both heavyweight members // These initializations are also in this interface. legacyRESTStorage, apiGroupInfo, err := legacyRESTStorageProvider.NewLegacyRESTStorage(restOptionsGetter) if err != nil { glog.Fatalf("Error building core storage: %v", err) } // The default is true to judge whether the controller is enable d. It has little to do with the theme here, but it is not in-depth for the time being. if c.EnableCoreControllers { serviceClient := coreclient.NewForConfigOrDie(c.GenericConfig.LoopbackClientConfig) bootstrapController := c.NewBootstrapController(legacyRESTStorage, serviceClient) if err := m.GenericAPIServer.AddPostStartHook("bootstrap-controller", bootstrapController.PostStartHook); err != nil { glog.Fatalf("Error registering PostStartHook %q: %v", "bootstrap-controller", err) } } // install core Group's API if err := m.GenericAPIServer.InstallLegacyAPIGroup(genericapiserver.DefaultLegacyAPIPrefix, &apiGroupInfo); err != nil { glog.Fatalf("Error in registering group versions: %v", err) } }
Let's first look at the interface NewLegacy RESTStorage () for creating APIGroupVersion and RESTStorage objects.
Path: pkg/registry/core/rest/storage_core.go
func (c LegacyRESTStorageProvider) NewLegacyRESTStorage(restOptionsGetter genericapiserver.RESTOptionsGetter) (LegacyRESTStorage, genericapiserver.APIGroupInfo, error) { // Initialize the creation of an APIGroup Version apiGroupInfo := genericapiserver.APIGroupInfo{ // The GroupMeta is retrieved from the structure initialized by APIRegistration Manager GroupMeta: *registered.GroupOrDie(api.GroupName), VersionedResourcesStorageMap: map[string]map[string]rest.Storage{}, // This api.Scheme has been introduced to initialization before. Scheme: api.Scheme, ParameterCodec: api.ParameterCodec, NegotiatedSerializer: api.Codecs, SubresourceGroupVersionKind: map[string]unversioned.GroupVersionKind{}, } // Determine if autoscaling is registered and enabled, and if so add to apiGroupInfo.SubresourceGroupVersionKind // key is the path of the resource if autoscalingGroupVersion := (unversioned.GroupVersion{Group: "autoscaling", Version: "v1"}); registered.IsEnabledVersion(autoscalingGroupVersion) { apiGroupInfo.SubresourceGroupVersionKind["replicationcontrollers/scale"] = autoscalingGroupVersion.WithKind("Scale") } var podDisruptionClient policyclient.PodDisruptionBudgetsGetter if policyGroupVersion := (unversioned.GroupVersion{Group: "policy", Version: "v1beta1"}); registered.IsEnabledVersion(policyGroupVersion) { apiGroupInfo.SubresourceGroupVersionKind["pods/eviction"] = policyGroupVersion.WithKind("Eviction") var err error podDisruptionClient, err = policyclient.NewForConfig(c.LoopbackClientConfig) if err != nil { return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err } } // Initialize a Legacy RESTStorage object // The following will be the initialization of each interface, there will be Node registration, IP application, NodePort application and so on. restStorage := LegacyRESTStorage{} // Creating Storage s podTemplateStorage := podtemplateetcd.NewREST(restOptionsGetter(api.Resource("podTemplates"))) eventStorage := eventetcd.NewREST(restOptionsGetter(api.Resource("events")), uint64(c.EventTTL.Seconds())) limitRangeStorage := limitrangeetcd.NewREST(restOptionsGetter(api.Resource("limitRanges"))) resourceQuotaStorage, resourceQuotaStatusStorage := resourcequotaetcd.NewREST(restOptionsGetter(api.Resource("resourceQuotas"))) secretStorage := secretetcd.NewREST(restOptionsGetter(api.Resource("secrets"))) serviceAccountStorage := serviceaccountetcd.NewREST(restOptionsGetter(api.Resource("serviceAccounts"))) persistentVolumeStorage, persistentVolumeStatusStorage := pvetcd.NewREST(restOptionsGetter(api.Resource("persistentVolumes"))) persistentVolumeClaimStorage, persistentVolumeClaimStatusStorage := pvcetcd.NewREST(restOptionsGetter(api.Resource("persistentVolumeClaims"))) configMapStorage := configmapetcd.NewREST(restOptionsGetter(api.Resource("configMaps"))) namespaceStorage, namespaceStatusStorage, namespaceFinalizeStorage := namespaceetcd.NewREST(restOptionsGetter(api.Resource("namespaces"))) restStorage.NamespaceRegistry = namespace.NewRegistry(namespaceStorage) endpointsStorage := endpointsetcd.NewREST(restOptionsGetter(api.Resource("endpoints"))) restStorage.EndpointRegistry = endpoint.NewRegistry(endpointsStorage) nodeStorage, err := nodeetcd.NewStorage(restOptionsGetter(api.Resource("nodes")), c.KubeletClientConfig, c.ProxyTransport) if err != nil { return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err } restStorage.NodeRegistry = node.NewRegistry(nodeStorage.Node) // Create PodStorage // api.Resource("pods") is a structure for synthesizing a GroupResource podStorage := podetcd.NewStorage( restOptionsGetter(api.Resource("pods")), nodeStorage.KubeletConnectionInfo, c.ProxyTransport, podDisruptionClient, ) serviceRESTStorage, serviceStatusStorage := serviceetcd.NewREST(restOptionsGetter(api.Resource("services"))) restStorage.ServiceRegistry = service.NewRegistry(serviceRESTStorage) var serviceClusterIPRegistry rangeallocation.RangeRegistry serviceClusterIPRange := c.ServiceIPRange if serviceClusterIPRange.IP == nil { return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, fmt.Errorf("service clusterIPRange is missing") } serviceStorageConfig, err := c.StorageFactory.NewConfig(api.Resource("services")) if err != nil { return LegacyRESTStorage{}, genericapiserver.APIGroupInfo{}, err } ServiceClusterIPAllocator := ipallocator.NewAllocatorCIDRRange(&serviceClusterIPRange, func(max int, rangeSpec string) allocator.Interface { mem := allocator.NewAllocationMap(max, rangeSpec) // TODO etcdallocator package to return a storage interface via the storageFactory etcd := etcdallocator.NewEtcd(mem, "/ranges/serviceips", api.Resource("serviceipallocations"), serviceStorageConfig) serviceClusterIPRegistry = etcd return etcd }) restStorage.ServiceClusterIPAllocator = serviceClusterIPRegistry var serviceNodePortRegistry rangeallocation.RangeRegistry ServiceNodePortAllocator := portallocator.NewPortAllocatorCustom(c.ServiceNodePortRange, func(max int, rangeSpec string) allocator.Interface { mem := allocator.NewAllocationMap(max, rangeSpec) // TODO etcdallocator package to return a storage interface via the storageFactory etcd := etcdallocator.NewEtcd(mem, "/ranges/servicenodeports", api.Resource("servicenodeportallocations"), serviceStorageConfig) serviceNodePortRegistry = etcd return etcd }) restStorage.ServiceNodePortAllocator = serviceNodePortRegistry controllerStorage := controlleretcd.NewStorage(restOptionsGetter(api.Resource("replicationControllers"))) serviceRest := service.NewStorage(restStorage.ServiceRegistry, restStorage.EndpointRegistry, ServiceClusterIPAllocator, ServiceNodePortAllocator, c.ProxyTransport) // Initialize a restStorage map and assign it to APIGroupInfo. Versioned Resources Storage Map ["v1"] restStorageMap := map[string]rest.Storage{ "pods": podStorage.Pod, "pods/attach": podStorage.Attach, "pods/status": podStorage.Status, "pods/log": podStorage.Log, "pods/exec": podStorage.Exec, "pods/portforward": podStorage.PortForward, "pods/proxy": podStorage.Proxy, "pods/binding": podStorage.Binding, "bindings": podStorage.Binding, "podTemplates": podTemplateStorage, "replicationControllers": controllerStorage.Controller, "replicationControllers/status": controllerStorage.Status, "services": serviceRest.Service, "services/proxy": serviceRest.Proxy, "services/status": serviceStatusStorage, "endpoints": endpointsStorage, "nodes": nodeStorage.Node, "nodes/status": nodeStorage.Status, "nodes/proxy": nodeStorage.Proxy, "events": eventStorage, "limitRanges": limitRangeStorage, "resourceQuotas": resourceQuotaStorage, "resourceQuotas/status": resourceQuotaStatusStorage, "namespaces": namespaceStorage, "namespaces/status": namespaceStatusStorage, "namespaces/finalize": namespaceFinalizeStorage, "secrets": secretStorage, "serviceAccounts": serviceAccountStorage, "persistentVolumes": persistentVolumeStorage, "persistentVolumes/status": persistentVolumeStatusStorage, "persistentVolumeClaims": persistentVolumeClaimStorage, "persistentVolumeClaims/status": persistentVolumeClaimStatusStorage, "configMaps": configMapStorage, "componentStatuses": componentstatus.NewStorage(componentStatusStorage{c.StorageFactory}.serversToValidate), } if registered.IsEnabledVersion(unversioned.GroupVersion{Group: "autoscaling", Version: "v1"}) { restStorageMap["replicationControllers/scale"] = controllerStorage.Scale } if registered.IsEnabledVersion(unversioned.GroupVersion{Group: "policy", Version: "v1beta1"}) { restStorageMap["pods/eviction"] = podStorage.Eviction } // Assign the restStorageMap above to v1 apiGroupInfo.VersionedResourcesStorageMap["v1"] = restStorageMap return restStorage, apiGroupInfo, nil }
After looking at this interface, let's go back and look at the m. GenericAPIServer. InstallLegacy APIGroup () interface:
Path: pkg/genericapiserver/genericapiserver.go
func (s *GenericAPIServer) InstallLegacyAPIGroup(apiPrefix string, apiGroupInfo *APIGroupInfo) error { // Determine whether the prefix parameter is correct if !s.legacyAPIGroupPrefixes.Has(apiPrefix) { return fmt.Errorf("%q is not in the allowed legacy API prefixes: %v", apiPrefix, s.legacyAPIGroupPrefixes.List()) } // Key interface, real install API if err := s.installAPIResources(apiPrefix, apiGroupInfo); err != nil { return err } // Get all version information under this Group // It should be used to find all current version information apiVersions := []string{} for _, groupVersion := range apiGroupInfo.GroupMeta.GroupVersions { apiVersions = append(apiVersions, groupVersion.Version) } // Install the version handler. // Add a handler at /<apiPrefix> to enumerate the supported api versions. apiserver.AddApiWebService(s.Serializer, s.HandlerContainer.Container, apiPrefix, func(req *restful.Request) *unversioned.APIVersions { clientIP := utilnet.GetClientIP(req.Request) apiVersionsForDiscovery := unversioned.APIVersions{ ServerAddressByClientCIDRs: s.discoveryAddresses.ServerAddressByClientCIDRs(clientIP), Versions: apiVersions, } return &apiVersionsForDiscovery }) return nil }
Then we continue to access the key interface S. install APIResources (apiPrefix, apiGroupInfo):
func (s *GenericAPIServer) installAPIResources(apiPrefix string, apiGroupInfo *APIGroupInfo) error { // Traverse through all GroupVersons under the Group for _, groupVersion := range apiGroupInfo.GroupMeta.GroupVersions { // Create APIGroup Version apiGroupVersion, err := s.getAPIGroupVersion(apiGroupInfo, groupVersion, apiPrefix) if err != nil { return err } if apiGroupInfo.OptionsExternalVersion != nil { apiGroupVersion.OptionsExternalVersion = apiGroupInfo.OptionsExternalVersion } // Install restful API based on the API Group Version you created earlier // The s.HandlerContainer.Container is the go-restful Container if err := apiGroupVersion.InstallREST(s.HandlerContainer.Container); err != nil { return fmt.Errorf("Unable to setup API %v: %v", apiGroupInfo, err) } } return nil } func (s *GenericAPIServer) getAPIGroupVersion(apiGroupInfo *APIGroupInfo, groupVersion unversioned.GroupVersion, apiPrefix string) (*apiserver.APIGroupVersion, error) { storage := make(map[string]rest.Storage) // If it is a core group, Version is "v1", and the initialization of the Versioned Resources Storage Map depends on // Previous New Legacy RESTStorage () interface, where initialization is performed // Traverse through all Resource Storages and assign values to storage for k, v := range apiGroupInfo.VersionedResourcesStorageMap[groupVersion.Version] { storage[strings.ToLower(k)] = v } // Create APIGroup Version version, err := s.newAPIGroupVersion(apiGroupInfo, groupVersion) // Set up Prefix and the core group is "/ api" version.Root = apiPrefix version.Storage = storage return version, err }
From API resources to restful API s, registration has been completed.
As for the apiGroupVersion.InstallREST() interface, let's start with a brief introduction, followed by another article with go-restful.
InstallREST() interface path: pkg/apiserver/apiserver.go
func (g *APIGroupVersion) InstallREST(container *restful.Container) error { installer := g.newInstaller() ws := installer.NewWebService() apiResources, registrationErrors := installer.Install(ws) lister := g.ResourceLister if lister == nil { lister = staticLister{apiResources} } AddSupportedResourcesWebService(g.Serializer, ws, g.GroupVersion, lister) container.Add(ws) return utilerrors.NewAggregate(registrationErrors) } func (a *APIInstaller) Install(ws *restful.WebService) (apiResources []unversioned.APIResource, errors []error) { errors = make([]error, 0) proxyHandler := (&ProxyHandler{ prefix: a.prefix + "/proxy/", storage: a.group.Storage, serializer: a.group.Serializer, mapper: a.group.Context, }) // Register the paths in a deterministic (sorted) order to get a deterministic swagger spec. paths := make([]string, len(a.group.Storage)) var i int = 0 for path := range a.group.Storage { paths[i] = path i++ } sort.Strings(paths) for _, path := range paths { // This interface is the key, eventually converting a rest.Storage object into an actual restful api, such as getter, lister, and other processing functions, and associating the actual URL s apiResource, err := a.registerResourceHandlers(path, a.group.Storage[path], ws, proxyHandler) if err != nil { errors = append(errors, fmt.Errorf("error in registering resource: %s, %v", path, err)) } if apiResource != nil { apiResources = append(apiResources, *apiResource) } } return apiResources, errors }
In this registration process, InstallREST finally calls the registerResourceHandlers() interface, which eventually converts a rest.Storage object into actual getter, lister, and other processing functions, and associates them with the actual URL.
User parameter configuration
Runtime-config: for enable/disable extensions group. By default, DaemonSets, Deployments, Horizontal Pod Autoscalers, Ingress, Jobs and ReplicaSets are enabled, and defaults under v1 are enabled. Additional functions can be set through this configuration. For example, disable deployments:--runtime-config=extensions/v1beta1/deployments=false.
Reference material
1.api-group.md: https://github.com/kubernetes...