preface

This blog mainly introduces Python's magic functions. In depth learning or python programming, python class writing will be more or less involved, including Python magic functions. For example, when writing a data loading generator, it may be involved__ next__，__ iter__ Function. Of course, the generator may be done with the keyword yield. Finally, in order to deepen the understanding of Python magic functions, this blog records some common Python magic functions in the form of code and description.

Magic function

definition

Magic methods are Python's built-in functions, usually starting with a double underscore. Each magic method corresponds to a built-in function or operator. For example, when len(obj) is used, obj is actually called__ len__ method. Therefore, when our object uses these methods, it is equivalent to rewriting or overloading these methods of the object.

Through dir(), you can view all the methods and properties of the object. The magic methods of the object start and end with double underscores. Take integer objects as an example:

Common magic methods

Common magic methods can be roughly divided into the following categories:

• Construction and initialization
• Class representation
• access control
• Comparison, operation, etc
• Container class operation
• Callable object
• serialize

Class construction and initialization

Class initialization generally involves three magic methods:__ init__，__ new__，__ del__.
When initializing a class, such as class_a = class_A(1), the first call is the__ new__ Method, return the instance object of the class, and then__ init__ Method to initialize the object.
__ new__ The method is as follows:

1. __ new__(cls,*args,**kwargs): there must be at least one parameter cls, representing the incoming class. This parameter is automatically provided by the Python interpreter during instantiation. If the object instance of this class is returned, the following parameters are directly passed to the__ init__.

class A:
    def __init__(self,a,b):
        print('this is A init')
        print(self)
        self.a=a
        self.b=b
    def __new__(cls, *args, **kwargs):
        print('this is A new')
        print('args:%s'%args)
        print('kwargs:%s'%kwargs)
        print(cls)
        print(object.__new__(cls))#<__ main__. A object at 0x000001bcd98fb3d0 >, an object instance of A
        return object.__new__(cls)#Create an instance and return
>>>a=A(1,b=10)
this is A new#Enter first__ new__
args:1
kwargs:{'b': 10}
<class '__main__.A'>
<__main__.A object at 0x000001BCD98FB3D0>
this is A init#Re entry__ init__
<__main__.A object at 0x000001D0BC3EB3D0>#self is__ new__ Returned object instance

1. __ new__ You can decide whether to use__ init__ Method, however, was executed__ new__， Not necessarily__ init__， Only__ new__ Returns the instance of the current class cls and the__ init__ To enter. Even if an instance of the parent class is returned, it must be an instance of the current class;

class A:
    def __init__(self,a,b):
        print('this is A init')
        self.a=a
        self.b=b
    def __new__(cls, *args, **kwargs):
        print('this is A new')
        print('args:%s'%args)
        print('kwargs:%s'%kwargs)
        print(cls)
>>>m=A(1,b=10)
this is A new
args:1
kwargs:{'b': 10}
<class '__main__.A'>
>>>print(m.a)#An error is reported and the current class is not entered__ init__ Initialize
AttributeError: 'NoneType' object has no attribute 'a'

1. object will__ new__ () method is defined as a static method, and at least one parameter cls needs to be passed. cls represents the class to be instantiated. This parameter is automatically provided by the Python interpreter during instantiation.
2. __ init__ () has a parameter self, which is__ new__ () returned instances

__ new__ Usage scenarios, such as singleton mode, factory mode, and inheritance of some immutable objects. This kind of application is very worthy of attention and use, which can greatly make the code look beautiful and concise;

__ del__ Method is a magic method called when the object is recycled by the system. It is called at the end of the object life cycle. Python uses automatic reference counting (ARC) to reclaim the space occupied by the object. When a variable in the program references the python object, python will automatically ensure that the object reference count is 1; When two variables in the program refer to the python object, python will automatically ensure that the reference count of the object is 2, and so on. If the reference count of an object becomes 0, it means that there are no variables in the program to refer to the object, indicating that the program no longer needs the object, so Python will recycle the object. So most of the time, we don't need to manually delete objects that are no longer used. Python's recycling mechanism will automatically help us do this.

Class representation

The magic methods related to the representation of class mainly include__ str__,__ repr__ And__ bool__

• __ str__ When printing an object print(obj), str(obj) is implicitly called, that is, the__ str__ method; Once the method is determined, it can be called through str(obj);
• __ repr__ The display of main expressions when directly outputting objects will be called__ repr__ method; Once this method is defined, it can be called through repr(obj).
• __ bool__: When bool(obj) is called, the__ bool__ () method, return True or False:

When there is no definition in the custom class__ str__ () and__ repr__ (), the default will be called when outputting the object__ str__ () and__ repr__ ()； When a class contains only__ str__ (), call the print() or str() function to output the object__ str__ (), direct output calls the default__ repr__ ()； When a class contains both__ str__ () also includes__ repr__ (), call the print() or str() function to output the object__ str__ (), direct output will call__ repr__ ()； When a class contains only__ repr__ (), call the print() or str() function for object output and direct output__ repr__ ().

Therefore, for custom classes, it is recommended to define__ str__ And__ repr__， Better interaction; Among them__ str__ It can be designed as a string to be converted and output, which provides some convenience when the subsequent str(obj) converts the object to a specific string output__ repr__ It can be designed to output more detailed information, such as column names and even some key parameter names, so as to facilitate the acquisition of accurate information of objects during development (for example, the machine learning module in sklearn is designed in this way)

Control attribute access

This kind of magic method mainly works when accessing, defining and modifying the attributes of the object. It mainly includes:

• __ getattr__(self, name): defines the behavior when a user attempts to obtain an attribute.
• __ getattribute__(self, name): define the behavior when the attribute of this class is accessed (call this method first to check whether the attribute exists. If it does not exist, then call _getattr _).
• __ setattr__(self, name, value): defines the behavior when an attribute is set.
• __ delattr__(self, name): defines the behavior when an attribute is deleted.

class A(object):
    def __init__(self,a,b):
        self.a=a
        self.b=b

    def __setattr__(self, key, value):
        print(key,value)
        print('this is magic method setattr')
    def __getattr__(self, item):
        print('getattr:%s'%item)
        print('this is magic method getattr')
    def __delattr__(self, item):
        print('delattr:%s'%item)
        print('this is magic method delattr')
        
>>>m=A(1,2)
a 1
this is magic method setattr#Called when initializing self.a=a__ setattr__
b 2
this is magic method setattr#Called when self.b=b is initialized__ setattr__

>>>a=m.a
getattr:a
this is magic method getattr#Called when accessing property a__ getattr__
>>>m.b=100
b 100
this is magic method setattr#Called when property b is modified__ setattr__
>>>delattr(m,'a')
delattr:a
this is magic method delattr#Called when property a is deleted__ delattr__
>>>print(m.a)
getattr:a
this is magic method getattr
None#Attribute a is deleted and is None

In the above code, the__ setattr__， Therefore, the overloaded function is called when the property is initialized__ setattr__； But when initializing a property call__ setattr__ It is necessary to cooperate with the attribute management of the instance__ dict__ That is, you need to set the attributes in self__ dict__ Otherwise, the instance cannot access these properties.

>>>print(m.a)
getattr:a
this is magic method getattr
None#As you can see, it did not initialize successfully to a=1 because of the overload__ setattr__ The property has not been placed inside the method__ dict__ Register in

#Modify the above__ setattr__:
class A(object):
    def __init__(self,a,b):
        self.a=a
        self.b=b
    def __setattr__(self, key, value):
        print(key,value)
        print('this is magic method setattr')
        self.__dict__[key] = value#In__ dict__ Register instance properties
        #super().__setattr__(key,value) can also be implemented by inheritance;

    def __getattr__(self, item):
        print('getattr:%s'%item)
        print('this is magic method getattr')
    def f(self):
        return self.__dict__#View attribute management dictionary
>>>m=A(1,2)
>>>m.a
1
>>>m.f()
{'a': 1, 'b': 2}

Usage scenarios for controlling attribute overloading: for example, when initializing an attribute, intercept the value of the attribute and make corresponding processing or judgment (such as type judgment or range judgment)

class A(object):
    def __init__(self,age,sex):
        self.age=age
        self.sex=sex

    def __setattr__(self, key, value):
        if key=='age':
            if not 0<=value<=100:
                raise Exception('age must between 0 and 100')
        elif key=='age':
            if not (value=='male' or value=='female'):
                raise Exception('sex must be male of female')
        else:
            pass
        super().__setattr__(key,value)
>>>m=A(age=102,sex='male')
Exception: age must between 0 and 100
>>>m=A(age=90,sex='hhh')
Exception: sex must be male of female
>>>m=A(age=90,sex='male')
>>>print(m.sex,m.age)
male 90

Comparison, operation, etc

By defining various magic methods related to comparison, operation and type, various comparison, operation and other operations between objects can be realized. There are many magic methods of this kind, which are not carried out one by one.

1. Magic function for comparison:
   
2. Binocular operator or function
   
3. Incremental operation
   
4. Type conversion
   

Container class operation

There are some methods to customize containers, such as python's built-in list,tuple,dict, etc; Containers are divided into variable containers and immutable containers. The details here need to understand the relevant protocols. If you customize an immutable container, you can only define__ len__ And__ getitem__； In addition to all magic methods of an immutable container, you also need to define__ setitem__ And__ delitem__； If the container can be iterated. You also need to define__ iter__

__ len__(self): returns the length of the container
__ getitem__(self,key): this method is called when the method of self[key] needs to be executed to call the object in the container
__ setitem__(self,key,value): this method is called when self[key] = value needs to be executed.
__ delitem__(self, key): this method needs to be called when del self[key] needs to be executed;
__ iter__(self): this method needs to be defined when the container can execute for x in container:, or use iter(container)
__ reversed__(self): implements the behavior when reversed() is called. The inverted version of the sequence should be returned. Implement a sequence only if it can be ordered, such as for a list or tuple.
__ contains__(self, item): defines the behavior generated when calling in and not in to test whether members exist.

class SpecialList(object):
    def __init__(self,values=None):
        if values is None:
            self.values=[]
        else:
            self.values=values
        self.count={}.fromkeys(range(len(self.values)),0)
    def __len__(self):#Access container length through len(obj)
        return len(self.values)

    def __getitem__(self, key):#Access the objects in the container through obj[key]
        self.count[key]+=1
        return self.values[key]

    def __setitem__(self, key, value):#Modify the objects in the container by obj[key]=value
        self.values[key]=value

    def __delitem__(self, key):#Delete the objects in the container through del obj[key]
        del self.values[key]

    def __iter__(self):#Traverse the container through the for loop
        return iter(self.values)
    
    def __next__(self):
        # Iteration details
        # If__ iter__ self must implement this method when returning
        if self._index >= len(self.values):
            raise StopIteration()
        value = self.values[self._index]
        self._index += 1
        return value

    def __reversed__(self):#reverse(obj) is used to reverse the objects in the container
        return SpecialList(reversed(self.values))

    def __contains__(self, item):#Judge whether the element is in the container by item in obj
        return item in self.values

    def append(self, value):
        self.values.append(value)

    def head(self):
        # Get the first element
        return self.values[0]

    def tail(self):
        # Gets all elements after the first element
        return self.values[1:]

Callable object

In Python, a method is also an advanced object. Through object implementation__ call__ You can call classes like calling methods.

class A(object):
    def __init__(self,a,b):
        self.a=a
        self.b=b
    def __call__(self,a):
        self.a=a
        
>>>m=A(1,2)
>>>m.a
1
>>>m.b
2
>>>id(m)
1460475565152
>>>m(100)#Calling a class directly like a function is essentially called__ call__ method
>>>m.a
100
>>>m.b
2
>>>id(m)
1460475565152

Application scenario:

1. Build a simple decorator (for example, the cached_property of the source code of the bottle framework)
2. It can be used as an abstract window function to abstract the use rules, rewrite other methods through subclasses, and achieve different results without changing the original code

serialize

It is also a built-in magic method called during serialization:

• __ getstate__ (): used for serialization of Python objects, specifying which information will be recorded during serialization
• __ setstate__ (): used for deserialization of Python objects, specifying how to use information when deserializing

class A(object):
    def __init__(self,a,b):
        self.a=a
        self.b=b
    def __getstate__(self):
        print('this is magic method __getstate__')
        return {'a':self.a,
                'b':self.b}#The state returned during serialization is the state passed in during deserialization
    def __setstate__(self, state):
        print('this is magic method __setstate__')
        self.a=state['a']
        self.b=300

import pickle
>>>a=A(1,2)
>>>a_1=pickle.dumps(a)#Call__ getstate__
>>>print(a_1)
this is magic method __getstate__
b'\x80\x04\x95&\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x01A\x94\x93\x94)\x81\x94}\x94(\x8c\x01a\x94K\x01\x8c\x01b\x94K\x02ub.'
>>>a_2=pickle.loads(a_1)#Call__ setstate__
>>>print(a_2)
this is magic method __setstate__
<__main__.A object at 0x000001BF5B086670>
>>>print(a_2.a,a_2.b)
1 300

summary

It is hoped that the magic function of python can be consciously and meaningfully used in the follow-up engineering work.