E-commerce project summary

E-commerce project (grain mall)

1, Project objectives:

Cereal mall is a B2C e-commerce project, which sells self operated goods to customers.

2, Project architecture

1. Technical architecture

The grain mall adopts the front and rear end separation development, which is a micro service project

Front end development with vue

The backend is developed with springboot + springcloud + springalibaba

For the business module, we use spring boot development

• Since there are calls between microservices between modules, we introduce openFeign to communicate between microservices

• Using feign requires service registration and service discovery, so we use nacos to register and discover micro services

• When our service goes online, it is very troublesome to change the configuration, so we can use nacos as the configuration center for unified configuration

• We use getWay gateway for dynamic routing and reverse proxy request filtering

• In case of service call timeout and service unavailability, or large user access to prevent service avalanche, we use sentinel for service fusing and service degradation

• Use sleuth and zipkin to track the service to see if the service is wrong and slow, so as to modify it

• Using oauth2 and spring security as authentication and authorization services

• Simplify our development with mybatisPlus

For data storage

• mysql is used for data persistence in order to make the service faster

• We use redis as cache

• Because we want to do full-text search and data analysis for the whole goods on the shelf, we use elastic search to do full-text search and data analysis

• Because it is a microservice architecture, there should be distributed transactions. We adopt flexible transactions + maximum notification + final consistency to realize distributed transactions

  The delay queue and dead letter queue of rabbitMQ are used for distributed transactions.

• We also introduced Alibaba cloud oss as the storage of images and objects

2. Service Architecture

The front end consists of two services: background management page and user page

Background we divide it into background management and commodity service inventory service user service order service shopping cart service retrieval service payment service central authentication service spike service

The front-end page sends http requests through the gateway first, and then the gateway filters and routes them to the specified service

1. Development of back-end management system:

1. We need to unify the encapsulation of returned results. It can be divided into success and failure

2. We also write a unified return exception information, which is divided into status code and exception reason. The status code is five digits. The first two digits represent module information, and the last three digits represent exception type (through enumeration)

/***
 * Error code and error message definition class
 * 1. The error code definition rule is 5 numbers
 * 2. The first two digits represent the business scenario, and the last three digits represent the error code. For example: 100001. 10: General 001: system unknown exception
 * 3. After maintaining the error code, you need to maintain the error description and define them as enumeration
 * Error code list:
 *  10: currency
 *      001: Parameter format verification
 *  11: commodity
 *  12: order
 *  13: Shopping Cart
 *  14: logistics
 */
public enum BizCodeEnume {
    UNKNOW_EXCEPTION(10000,"System unknown exception"),
    VAILD_EXCEPTION(10001,"Parameter format verification failed"),
    PRODUCT_EXCEPTION(11000,"Abnormal listing of goods");

    private int code;
    private String msg;
    BizCodeEnume(int code,String msg){
        this.code = code;
        this.msg = msg;
    }

    public int getCode() {
        return code;
    }

    public String getMsg() {
        return msg;
    }
}

1. Unified global exception handling

   Use RestControllerAdvice to operate the controller uniformly

   Add handlerException annotation for unified exception handling

   @Slf4j
   @RestControllerAdvice(basePackages = "com.atguigu.gulimall.product.controller")
   public class GulimallExceptionControllerAdvice {
   
   
       @ExceptionHandler(value= MethodArgumentNotValidException.class)
       public R handleVaildException(MethodArgumentNotValidException e){
           log.error("There is a problem with data verification{}，Exception type:{}",e.getMessage(),e.getClass());
           BindingResult bindingResult = e.getBindingResult();
   
           Map<String,String> errorMap = new HashMap<>();
           bindingResult.getFieldErrors().forEach((fieldError)->{
               errorMap.put(fieldError.getField(),fieldError.getDefaultMessage());
           });
           return R.error(BizCodeEnume.VAILD_EXCEPTION.getCode(),BizCodeEnume.VAILD_EXCEPTION.getMsg()).put("data",errorMap);
       }
   
       @ExceptionHandler(value = Throwable.class)
       public R handleException(Throwable throwable){
   
           log.error("Error:",throwable);
           return R.error(BizCodeEnume.UNKNOW_EXCEPTION.getCode(),BizCodeEnume.UNKNOW_EXCEPTION.getMsg());
       }
   
   
   }
   

Database structure commodity library inventory Library

The inventory table needs to show how many items there are

A commodity has two attributes: sku and spu. spu describes a kind of commodity and sku describes a kind of commodity. Huawei mobile phone is spu Huawei mobile phone sapphire blue 8+64G fashion mobile phone is his sku.

Therefore, the database is designed from the two aspects of spu and sku, which will cover all the information of goods. spuInfo skuInfo spuAttr skuAttr images

Firstly, it compiles the attributes of three-level commodity classification sku and spu, and classifies sku, spu and three-level commodity shelf function and member management

The function of putting goods on the shelf is divided into filling in basic information, filling in specification parameters, sales attributes (spu) and entering sku. Finally, it is saved to the commodity library and inventory library

The user module needs to write whether the member level growth experience is mail free or disabled.

All their requests are sent by vue to reach the background service through the gateway

2. Back end page

3. Performance optimization

Product details page

First display the three-level classification. Query the display of goods according to the three-level classification. Here, you need to switch between sales attributes. Query all SKUs according to the id of spu, and then display

The id and spu of the bag of sku can get the details of the goods.

The product details page is displayed to users as soon as they enter, so there will be a large number of visits and high concurrency. So we need to optimize the system,

• Optimization should start from sql and be divided into two types

One is to optimize sql, use indexes as much as possible, reduce the rows and columns of queries, and reduce the associated sub databases and sub tables of large tables to reduce the pressure on a single point

Second, reduce the number of database queries. We can introduce caching and use redis as caching. When displaying data, we first cache the query data. If there is no data in the cache, we query the data in the database, and the queried data is added to the cache. If there is data in the cache, it is returned directly.

4. Full text search

In the commodity search service, we need to search and query many attributes, such as commodity attribute, price function, screen size, length, cpu, memory camera, etc. at this time, mysql can not adapt to this function. We have introduced ElasticSearch for full-text search and data analysis.

When we save the elasticSearch, we save the information of the spu, because the sku is too large. The spu is the sum of a class of goods.

Query the basic information and product specification inventory information of spu products according to spuid, and store them in elastic search.

5. Cache

• Cache should be stored in time, and data consistency is not required

• Read write less data

data = cache.load(info); //Load data from cache
	if(data == null){
        data = db.load(info);  //Load data from database
        cache.put(id,data,ttl); //Put it in the cache and set the expiration time
    }
return data;

The expiration time must be set for the data in the cache to avoid that the data is not actively updated or can trigger automatic update to avoid data crash and inconsistency forever.

Store json data in the cache (recommended), because json is easy to read across platforms

• Caching has three problems

• Buffer breakdown

• Cache avalanche

• Cache penetration

  • Cache breakdown is a large number of requests to query a data (hotspot data). If the key of this data expires, query the database

  • Cache avalanche means that a batch of data fails at the same time. A large number of requests to query the data enter the database, resulting in the unavailability of the system

  • The cache penetrates a large number of requests to query a nonexistent data. You need to do io queries in the database all the time to increase the performance consumption of the system.

    Buffer breakdown plus distributed lock

    Cache avalanche randomly set expiration time

    Cache through empty results, and set a very short expiration time

• Cache inconsistency

  • When modifying the database in double write mode, modify the cache
  • Delete cache when modifying database in failure mode
  • Add expiration time to cache
  • Add read / write lock

6. Distributed lock

redisson is used for distributed locking.

When many requests query a commodity, because there is no in the cache at this time, they will query in the database. At this time, the database is under great pressure and consumes a lot of resources.

We can lock him at this time. Everyone gets the lock when they come in. Whoever gets it will go to the database, and those who don't get it will continue to wait. This solves the problem of cache penetration.

• Reentrant Lock

When there is no lock, everyone grabs the lock. Whoever grabs it is who owns it

The watchdog mechanism is implemented inside redisson. If the redisson instance is not closed, the lock time will be automatically extended for 30s

// Automatically unlock 10 seconds after locking
// There is no need to call the unlock method to unlock manually
lock.lock(10, TimeUnit.SECONDS);

// Try to lock, wait for 100 seconds at most, and unlock automatically 10 seconds after locking
boolean res = lock.tryLock(100, 10, TimeUnit.SECONDS);
if (res) {
   try {
     ...
   } finally {
       lock.unlock();
   }
}

• fair lock

  When there are multiple threads to acquire the lock, priority will be given to the thread that makes the request first. If the thread goes down, redisson will wait 5s to try another thread.

  First come, first served

  RLock fairLock = redisson.getFairLock("anyLock");
  // Most common usage
  fairLock.lock();
  

• Fair lock with timeout

  // Automatic unlocking after 10 seconds
  // There is no need to call the unlock method to unlock manually
  fairLock.lock(10, TimeUnit.SECONDS);
  
  // Try to lock, wait for 100 seconds at most, and unlock automatically 10 seconds after locking
  boolean res = fairLock.tryLock(100, 10, TimeUnit.SECONDS);
  ...
  fairLock.unlock();
  

• Asynchronous fair lock

  RLock fairLock = redisson.getFairLock("anyLock");
  fairLock.lockAsync();
  fairLock.lockAsync(10, TimeUnit.SECONDS);
  Future<Boolean> res = fairLock.tryLockAsync(100, 10, TimeUnit.SECONDS);
  

• Interlock (MultiLock)

  All locks are locked successfully

  RLock lock1 = redissonInstance1.getLock("lock1");
  RLock lock2 = redissonInstance2.getLock("lock2");
  RLock lock3 = redissonInstance3.getLock("lock3");
  
  RedissonMultiLock lock = new RedissonMultiLock(lock1, lock2, lock3);
  // Simultaneous locking: lock1 lock2 lock3
  // All locks are locked successfully.
  lock.lock();
  ...
  lock.unlock();
  

  Timeout

RedissonMultiLock lock = new RedissonMultiLock(lock1, lock2, lock3);
// Lock lock1, lock2 and lock3. If they are not unlocked manually, they will be unlocked automatically after 10 seconds
lock.lock(10, TimeUnit.SECONDS);

// Wait for 100 seconds for locking and unlock automatically after 10 seconds of successful locking
boolean res = lock.tryLock(100, 10, TimeUnit.SECONDS);
...
lock.unlock();

• red lock

  Multiple services, one user can only have one request at the same time

  RLock lock1 = redissonInstance1.getLock("lock1");
  RLock lock2 = redissonInstance2.getLock("lock2");
  RLock lock3 = redissonInstance3.getLock("lock3");
  
  RedissonRedLock lock = new RedissonRedLock(lock1, lock2, lock3);
  // Simultaneous locking: lock1 lock2 lock3
  // Red locks are successful when they are successfully locked on most nodes.
  lock.lock();
  ...
  lock.unlock();
  

• Read write lock

  Read / write locks allow multiple read locks and one write lock

  Read locks are shared locks and write locks are exclusive locks

  RReadWriteLock rwlock = redisson.getReadWriteLock("anyRWLock");
  // Most common usage
  rwlock.readLock().lock();
  // or
  rwlock.writeLock().lock();
  

  // Automatic unlocking after 10 seconds
  // There is no need to call the unlock method to unlock manually
  rwlock.readLock().lock(10, TimeUnit.SECONDS);
  // or
  rwlock.writeLock().lock(10, TimeUnit.SECONDS);
  
  // Try to lock, wait for 100 seconds at most, and unlock automatically 10 seconds after locking
  boolean res = rwlock.readLock().tryLock(100, 10, TimeUnit.SECONDS);
  // or
  boolean res = rwlock.writeLock().tryLock(100, 10, TimeUnit.SECONDS);
  ...
  lock.unlock();
  

• Semaphore

  An initial value can be set for semaphores. All threads can obtain semaphores. One thread consumes one semaphore. If the semaphore is negative, the semaphore cannot be obtained. After the thread executes, the semaphore should be released

  RSemaphore semaphore = redisson.getSemaphore("semaphore");
  semaphore.acquire();
  //or
  semaphore.acquireAsync();
  semaphore.acquire(23);
  semaphore.tryAcquire();
  //or
  semaphore.tryAcquireAsync();
  semaphore.tryAcquire(23, TimeUnit.SECONDS);
  //or
  semaphore.tryAcquireAsync(23, TimeUnit.SECONDS);
  semaphore.release(10);
  semaphore.release();
  //or
  semaphore.releaseAsync();
  

7. Asynchronous

We want to get the detailed information of the product, the introduction of spu, the specifications and parameters of spu, whether to participate in the discount or not. These are queried, encapsulated with an object, and then returned

Introduction to spu whether the specification parameters of spu participate in the discount depends on the basic information of the commodity, while the picture can be queried directly. The specification parameters of spu and whether to participate in the discount spu introduction do not affect each other. Therefore, we can arrange them asynchronously to make the user experience better.

/**
 * @Description: Thread pool configuration class
 **/

@EnableConfigurationProperties(ThreadPoolConfigProperties.class)
@Configuration
public class MyThreadConfig {


    @Bean
    public ThreadPoolExecutor threadPoolExecutor(ThreadPoolConfigProperties pool) {
        //Number of core threads, maximum number of threads, idle thread lifetime, time unit, thread blocking queue, factory for creating threads, rejection policy
        return new ThreadPoolExecutor(
                pool.getCoreSize(),
                pool.getMaxSize(),
                pool.getKeepAliveTime(),
                TimeUnit.SECONDS,
                new LinkedBlockingDeque<>(100000),
                Executors.defaultThreadFactory(),
                new ThreadPoolExecutor.AbortPolicy()
        );
    }

}

 @Override
    public SkuItemVo item(Long skuId) throws ExecutionException, InterruptedException {

        SkuItemVo skuItemVo = new SkuItemVo();

        CompletableFuture<SkuInfoEntity> infoFuture = CompletableFuture.supplyAsync(() -> {
            //1. Acquisition of SKU basic information pms_sku_info
            SkuInfoEntity info = this.getById(skuId);
            skuItemVo.setInfo(info);
            return info;
        }, executor);


        CompletableFuture<Void> saleAttrFuture = infoFuture.thenAcceptAsync((res) -> {
            //3. Get the sales attribute combination of spu
            List<SkuItemSaleAttrVo> saleAttrVos = skuSaleAttrValueService.getSaleAttrBySpuId(res.getSpuId());
            skuItemVo.setSaleAttr(saleAttrVos);
        }, executor);


        CompletableFuture<Void> descFuture = infoFuture.thenAcceptAsync((res) -> {
            //4. Get introduction PMS of SPU_ spu_ info_ desc
            SpuInfoDescEntity spuInfoDescEntity = spuInfoDescService.getById(res.getSpuId());
            skuItemVo.setDesc(spuInfoDescEntity);
        }, executor);


        CompletableFuture<Void> baseAttrFuture = infoFuture.thenAcceptAsync((res) -> {
            //5. Get the specification parameter information of spu
            List<SpuItemAttrGroupVo> attrGroupVos = attrGroupService.getAttrGroupWithAttrsBySpuId(res.getSpuId(), res.getCatalogId());
            skuItemVo.setGroupAttrs(attrGroupVos);
        }, executor);


        // Long spuId = info.getSpuId();
        // Long catalogId = info.getCatalogId();

        //2. Picture information PMS of SKU_ sku_ images
        CompletableFuture<Void> imageFuture = CompletableFuture.runAsync(() -> {
            List<SkuImagesEntity> imagesEntities = skuImagesService.getImagesBySkuId(skuId);
            skuItemVo.setImages(imagesEntities);
        }, executor);

        CompletableFuture<Void> seckillFuture = CompletableFuture.runAsync(() -> {
            //3. Remotely call to query whether the current sku participates in the second kill discount
            R skuSeckilInfo = seckillFeignService.getSkuSeckilInfo(skuId);
            if (skuSeckilInfo.getCode() == 0) {
                //query was successful
                SeckillSkuVo seckilInfoData = skuSeckilInfo.getData("data", new TypeReference<SeckillSkuVo>() {
                });
                skuItemVo.setSeckillSkuVo(seckilInfoData);

                if (seckilInfoData != null) {
                    long currentTime = System.currentTimeMillis();
                    if (currentTime > seckilInfoData.getEndTime()) {
                        skuItemVo.setSeckillSkuVo(null);
                    }
                }
            }
        }, executor);


        //Wait until all the tasks are completed
        CompletableFuture.allOf(saleAttrFuture,descFuture,baseAttrFuture,imageFuture,seckillFuture).get();

        return skuItemVo;
    }

thenApply receives the return value of the previous thread and has its own return value

thenAccept receives the return value of the previous thread. It has no return value

ThenRun does not accept the previous return value and runs itself directly

8. Certification services

We have done the unified authentication login function. Use spring security to control authentication and login.

9. Shopping Cart Service

We save the goods in redis and select Map < string, map < string, string > > as the data structure

Each user should have its own independent shopping cart, with user id as key and commodity information as value. Each user can crud the content in the shopping cart, so value is also a map, with the commodity id as the key and the commodity parameter as the value

10. Message queue rabbitMQ

There are three confirmation mechanisms for rabbitMQ messages: confirmCallback, returnCallBack and ack

confirmCallBack is a callback triggered when a message is sent to the switch

returnCallBack is the callback triggered when the message reaches the queue

ack is a callback triggered by the message from the queue to the consumer

Ack() acknowledgement message will be deleted by the queue

nack() message processing error messages will be processed by others and can be processed in batches

reject() error messages will be handled by others

After the customer places an order, the order status has unpaid, paid, shipped, received and cancelled the fifth middle school

If the customer has not paid within 3 hours after placing the order, we will not place the order for it. We should cancel the order and reduce the inventory. When placing the order, we can send a message to enter the delay queue. If no one handles the message after it expires, it means that the customer has cancelled the order. At this time, we can listen to the messages in the dead letter queue. If the employees in the dead letter queue have messages, we will unlock the inventory according to the messages in the dead letter queue.

11. Interface idempotency

The effect of one-time interface execution is the same as that of multiple executions (for example, we can't limit the deduction of payment)

• token mechanism solves interface idempotency

When we execute the business, we can generate a token, store the token in redis and put it in the request header. When requesting the interface, we obtain the token and query redis. If there is a token in redis, it is the first request. We delete the token in redis and execute the business. If there is no token, the interface is requested many times, and we directly return the duplicate tag node client

token  =  server.createToken
    redis.put(payId,token)
    request.header.add(payId,token)
    
    token = request.header.get(payId)
    if(token.equals(redis.get(payId))){
        try{
            Execute business;
            redis.del(payId);   //The acquisition, comparison and deletion of token s should be atomic and can be implemented by lua script
        	return data;
        }
        catch(error){
            redis.put(payId,token);  //Delete the token first. If the business execution fails, set the token here to request
             Re request
        }
       
    }
	return repeatMark

"if redis.call('get',KEYS[1]) == ARGV[1] then return redis.call('del',KEYS[1]) else return 0 end"     //The acquisition, comparison and deletion of token s should be atomic and can be implemented by lua script

Optimistic lock of database

Distributed lock

set anti duplication of redis

MD5 = Generate from data MD5 
redis.sadd(MD5)

MD5 = request.header.get(MD5)
if(redis.sadd(MD5)){
	redis.srem(MD5)
  First request
  Execute business
}

Save the globally unique request ID to redis (set)

12. Services

Four characteristics (acid) atomicity consistency isolation persistence

Four isolation levels read uncommitted (dirty read) read committed (non repeatable read) repeatable (phantom read) serialization

Propagation mechanism of transaction

propagation required if there is no transaction, create a transaction. If there is a transaction, join the transaction

propagation support if there is a transaction, join the transaction. If there is no transaction, execute it in a non transactional manner

If there is a transaction, join the transaction. If there is no transaction, throw an exception

Propagation requirements news creates a new transaction anyway

propagation no support does not support suspending transactions when they exist

The propagation never runs in a non transactional manner,

13. Distributed transactions

CAP principle consistency availability partition tolerance partition tolerance

The CAP principle can only support two, not three. Using RAFT algorithm

Raft (thesecretlivesofdata.com)

Based on base theory

We adopt the ap principle to achieve weak consistency in distributed transactions

Final consistency of basic available soft state

We use the flexible transaction based on base theory = = reliable message + best effort notification + final consistency scheme

14. Second kill service

Because the second kill has a very large traffic, we should put the goods on the shelves in advance into the cache

• The single responsibility principle and independent deployment are highly concurrent services, so we should follow the single responsibility principle and deploy independently

We have opened a new micro service to deal with second kill business

• Second kill links should be encrypted

  Prevent someone from participating in the second kill before it starts

• Inventory preheating, quick deduction

  We should not deduct inventory during the second kill. We should set a semaphore whose value is the amount of inventory

  The semaphore is over, and the second kill is over

• If the malicious request is intercepted (at the gateway) more than 10 times a second, we will judge it as a malicious request

• For traffic peak shaving, we can set the verification code mechanism to allow users to access the verification code during the second kill, and disperse the requests at various time nodes

• When the current limiting fuse downgrades to 1 million requests in one second, we let him wait for flower girl 2s to receive the request; When the service call takes a long time, we let it fail quickly instead of waiting there; If the traffic is too large, we can direct part of the traffic to a degraded page

• When the queue peak shaving request comes in, we ask him to grab the semaphore, grab the semaphore, and then send a message to the queue to let him execute the second kill service

Scheduled task @ schedule (seconds, minutes, hours, days, months, weeks) write either day or week? All right

• Goods on the shelf

We can use scheduled tasks to put goods on the shelves

Query seckill information (activity id, activity start time, activity end time list < Product >)

There are two maps in redis

A map key stores the start time and end time, and value is stored as a commodity id

the other one Map storage key Is the number of commodity events__id，value  Details of inventory items

We should set a token in the product information, which can prevent some people from attacking seckill products with malicious scripts

When they come to request, they must bring this token to request

We should set a distributed semaphore for the goods to be killed second. The semaphore is the goods Skell+stock+token, and the value is the inventory of the goods.

When products are on the shelf, we should ensure the idempotency of the interface

We can set the distributed lock when the scheduled task is opened to obtain the locked services and goods on the shelf

When adding a product to redis, we can judge that the key already exists. If it does not exist, we will set the product

• Query items for activity time

Send the information of the second kill order (order number, commodity information, member information) to MQ. The order service monitors the queue of MQ and gets the information of the second kill order to create an order for the user.

You need to deduct inventory when creating an order, but you can also cancel the order; It is also possible that after the inventory deduction is successful, the remote call fails due to network reasons and the order is rolled back.

• We create an order. If the order is not paid within 30 minutes, we think it has been closed

After we lock the inventory for 40 minutes, if the order has not been paid, we should unlock the inventory

• When creating an order, call inventory and send a message to MQ (order number);

The customs service listens to MQ, obtains the order number, and queries the order status. If the order status is unpaid, the order will be closed; Then send a message to MQ that the order has been closed,

The unlock inventory service listens to this queue and unlocks the inventory after receiving the customs clearance message

• Locking inventory is a transaction

If the inventory is locked successfully, send a message to MQ (inventory locking detail sheet id)

The inventory unlocking service listens to MQ to get the inventory detail id, query the inventory id, and query the inventory detail document according to the inventory id. if not, the inventory has not been locked successfully, and there is no need to unlock; If there is inventory, query the order status according to the order id. if there is an order and the order status is cancelled, unlock the inventory. If it has been paid, there is no need to unlock (handle the manual ack mechanism; Ensure message reliability (delivery)