Summary: Very interesting operation.

• Original: Realization of web Page Recording Screen
• Translator: frontdog

Fundebug Copyright shall be owned by the original author upon authorization to reproduce.

Written in front

After reading the comments, I suddenly realized that I didn't explain it in advance. This article can be said to be a research and learning article. It is a feasible set of programs for me. I will read some similar codebases that have been opened up in the future to fill in some details that I have missed. So we can use them carefully as learning articles and production environments.

Video Reproduction Error Scene

If your application is connected to the web apm system, you may know that the apm system can help you catch uncovered errors on the page, give the error stack, and help you locate BUG. However, sometimes, when you do not know the user's specific operation, there is no way to reproduce this error. At this time, if there is an operation recording screen, you can clearly understand the user's operation path, thus reproducing the BUG and fixing it.

Ideas for Realization

Idea 1: Use Canvas screenshots

This idea is relatively simple, that is to use canvas to draw web page content, the more famous library is: html2canvas The simple principle of this library is:

1. Collect all DOM s and store them in a queue.
2. According to zIndex, DOM is drawn one by one on Canvas with its CSS style through certain rules.

This implementation is complex, but we can use it directly, so we can get the screenshots of the page we want.

In order to make the generated video more smoothly, we need to generate about 25 frames per second, that is, 25 screenshots. The flow chart is as follows:

However, this idea has one fatal disadvantage: in order to make the video smooth, we need 25 pictures in a second, a figure 300 KB. When we need 30 seconds of video, the total size of the picture is 220 M, so the network overhead is obviously not good.

Idea 2: Record all operations to reproduce

In order to reduce network overhead, let's change our thinking. We record the next steps on the basis of the first page. When we need to play, we apply these operations in order so that we can see the changes of the page. This idea separates mouse operation from DOM changes:

Mouse changes:

1. Listen for mouseover events and record clientX and clientY of the mouse.
2. When replaying, use js to draw a false mouse and change the position of the mouse according to the coordinate record.

DOM changes:

1. Take a full snapshot of the page DOM. It includes collecting styles, removing JS scripts, and marking each DOM element with an id through certain rules.
2. Listen for all events that may affect the interface, such as mouse events, input events, scroll events, zoom events, etc. Each event records parameters and target elements. The target element can be the id just recorded. Each change event can be recorded as an incremental snapshot.
3. Send a certain number of snapshots to the back end.
4. Play in the background according to the snapshot and operation chain.

Of course, this description is relatively simple, the mouse record is relatively simple, we do not start to talk about, mainly explain the DOM monitoring implementation ideas.

First full snapshot of the page

First of all, you might think that to take a full snapshot of a page, you can use outerHTML directly.

const content = document.documentElement.outerHTML;

This simply records all the DOM of the page. You just need to add the tag id to the DOM first, then get the outerHTML, and then remove the JS script.

However, there is a problem here. DOM recorded with outerHTML will merge two adjacent TextNodes into one node, and Mutation Observer will be used when we monitor DOM changes later. At this time, you need a lot of processing to be compatible with the merging of TextNodes, otherwise you will not be able to locate the object of the operation when restoring the operation. Standard node.

So, do we have a way to maintain the original structure of the page DOM?

The answer is yes. Here we use the Virtual DOM to record the DOM structure, turn the document Element into the Virtual DOM, record it, and then regenerate the DOM when we restore it later.

Converting DOM into Virtual DOM

We need only care about two Node types here: Node.TEXT_NODE and Node.ELEMENT_NODE. At the same time, it should be noted that the creation of SVG and SVG sub-elements requires the use of API: create ElementNS, so when we record the Virtual DOM, we need to pay attention to the namespace record, code:

const SVG_NAMESPACE = 'http://www.w3.org/2000/svg';
const XML_NAMESPACES = ['xmlns', 'xmlns:svg', 'xmlns:xlink'];

function createVirtualDom(element, isSVG = false)  {
  switch (element.nodeType) {
    case Node.TEXT_NODE:
      return createVirtualText(element);
    case Node.ELEMENT_NODE:
      return createVirtualElement(element, isSVG || element.tagName.toLowerCase() === 'svg');
    default:
      return null;
  }
}

function createVirtualText(element) {
  const vText = {
    text: element.nodeValue,
    type: 'VirtualText',
  };
  if (typeof element.__flow !== 'undefined') {
    vText.__flow = element.__flow;
  }
  return vText;
}

function createVirtualElement(element, isSVG = false) {
  const tagName = element.tagName.toLowerCase();
  const children = getNodeChildren(element, isSVG);
  const { attr, namespace } = getNodeAttributes(element, isSVG);
  const vElement = {
    tagName, type: 'VirtualElement', children, attributes: attr, namespace,
  };
  if (typeof element.__flow !== 'undefined') {
    vElement.__flow = element.__flow;
  }
  return vElement;
}

function getNodeChildren(element, isSVG = false) {
  const childNodes = element.childNodes ? [...element.childNodes] : [];
  const children = [];
  childNodes.forEach((cnode) => {
    children.push(createVirtualDom(cnode, isSVG));
  });
  return children.filter(c => !!c);
}

function getNodeAttributes(element, isSVG = false) {
  const attributes = element.attributes ? [...element.attributes] : [];
  const attr = {};
  let namespace;
  attributes.forEach(({ nodeName, nodeValue }) => {
    attr[nodeName] = nodeValue;
    if (XML_NAMESPACES.includes(nodeName)) {
      namespace = nodeValue;
    } else if (isSVG) {
      namespace = SVG_NAMESPACE;
    }
  });
  return { attr, namespace };
}

With the above code, we can convert the entire document Element into Virtual DOM, where _flow is used to record some parameters, including tag ID, etc. Virtual Node records: type, attributes, children, namespace.

Reduction of Virtual DOM to DOM

It's easy to restore Virtual DOM to DOM by creating DOM recursively, where nodeFilter is used to filter script elements because we don't need JS script execution.

function createElement(vdom, nodeFilter = () => true) {
  let node;
  if (vdom.type === 'VirtualText') {
    node = document.createTextNode(vdom.text);
  } else {
    node = typeof vdom.namespace === 'undefined'
      ? document.createElement(vdom.tagName)
      : document.createElementNS(vdom.namespace, vdom.tagName);
    for (let name in vdom.attributes) {
      node.setAttribute(name, vdom.attributes[name]);
    }
    vdom.children.forEach((cnode) => {
      const childNode = createElement(cnode, nodeFilter);
      if (childNode && nodeFilter(childNode)) {
        node.appendChild(childNode);
      }
    });
  }
  if (vdom.__flow) {
    node.__flow = vdom.__flow;
  }
  return node;
}

DOM Structure Change Monitoring

Here, we use API: Mutation Observer, and it's even more gratifying that this API is compatible with all browsers, so we can use it boldly.

Using Mutation Observer:

const options = {
  childList: true, // Whether to observe the changes of subnodes
  subtree: true, // Whether to observe the changes of all descendant nodes
  attributes: true, // Whether to observe changes in attributes
  attributeOldValue: true, // Whether to observe the old values of the changes in attributes
  characterData: true, // Whether the content of a node or the text of a node changes
  characterDataOldValue: true, // Is the old value of a change in node content or text?
  // AttributeFilter: ['class','src'] will be ignored when attributes do not change in this array
};

const observer = new MutationObserver((mutationList) => {
    // mutationList: array of mutation
});
observer.observe(document.documentElement, options);

It's easy to use. You only need to specify a root node and some options to monitor. When DOM changes, there will be a mutation List in the callback function, which is a list of changes in DOM. The structure of mutation is roughly as follows:

{
    type: 'childList', // or characterData,attributes
    target: <DOM>,
    // other params
}

We use an array to store mutation s. The specific callback is:

const onMutationChange = (mutationsList) => {
  const getFlowId = (node) => {
    if (node) {
      // The newly inserted DOM has no tags, so compatibility is required here.
      if (!node.__flow) node.__flow = { id: uuid() };
      return node.__flow.id;
    }
  };
  mutationsList.forEach((mutation) => {
    const { target, type, attributeName } = mutation;
    const record = { 
      type, 
      target: getFlowId(target), 
    };
    switch (type) {
      case 'characterData':
        record.value = target.nodeValue;
        break;
      case 'attributes':
        record.attributeName = attributeName;
        record.attributeValue = target.getAttribute(attributeName);
        break;
      case 'childList':
        record.removedNodes = [...mutation.removedNodes].map(n => getFlowId(n));
        record.addedNodes = [...mutation.addedNodes].map((n) => {
          const snapshot = this.takeSnapshot(n);
          return {
            ...snapshot,
            nextSibling: getFlowId(n.nextSibling),
            previousSibling: getFlowId(n.previousSibling)
          };
        });
        break;
    }
    this.records.push(record);
  });
}

function takeSnapshot(node, options = {}) {
  this.markNodes(node);
  const snapshot = {
    vdom: createVirtualDom(node),
  };
  if (options.doctype === true) {
    snapshot.doctype = document.doctype.name;
    snapshot.clientWidth = document.body.clientWidth;
    snapshot.clientHeight = document.body.clientHeight;
  }
  return snapshot;
}

Just note that when you deal with the new DOM, you need an incremental snapshot. Virtual DOM is still used to record it here, and DOM is still generated and inserted into the parent element when playing back. So you need to refer to DOM, which is the sibling node.

Form Element Monitoring

The Automation Observer above can't monitor the value changes of elements such as input, so we need to deal with the values of form elements in a special way.

oninput event monitoring

MDN documents: developer.mozilla.org/en-US/docs/...

Event objects: select, input, textarea

window.addEventListener('input', this.onFormInput, true);

onFormInput = (event) => {
  const target = event.target;
  if (
    target && 
    target.__flow &&
    ['select', 'textarea', 'input'].includes(target.tagName.toLowerCase())
   ) {
     this.records.push({
       type: 'input', 
       target: target.__flow.id, 
       value: target.value, 
     });
   }
}

Capture is used to capture events in window s, which is also handled later. The reason for this is that we can and often prevent bubbles in the bubbling phase to achieve some functions, so capturing can reduce the loss of events. In addition, scroll events are not bubbling, so capturing must be used.

onchange event monitoring

MDN documents: developer.mozilla.org/en-US/docs/...

input events cannot satisfy the monitoring of type for checkbox and radio s, so onchange events are needed to monitor them.

window.addEventListener('change', this.onFormChange, true);

onFormChange = (event) => {
  const target = event.target;
  if (target && target.__flow) {
    if (
      target.tagName.toLowerCase() === 'input' &&
      ['checkbox', 'radio'].includes(target.getAttribute('type'))
    ) {
      this.records.push({
        type: 'checked', 
        target: target.__flow.id, 
        checked: target.checked,
      });
    }
  }
}

onfocus event monitoring

MDN documents: developer.mozilla.org/en-US/docs/...

window.addEventListener('focus', this.onFormFocus, true);

onFormFocus = (event) => {
  const target = event.target;
  if (target && target.__flow) {
    this.records.push({
      type: 'focus', 
      target: target.__flow.id,
    });
  }
}

onblur event monitoring

MDN documents: developer.mozilla.org/en-US/docs/...

window.addEventListener('blur', this.onFormBlur, true);

onFormBlur = (event) => {
  const target = event.target;
  if (target && target.__flow) {
    this.records.push({
      type: 'blur', 
      target: target.__flow.id,
    });
  }
}

Media Element Change Monitoring

This refers to audio and video, similar to the form elements above, which can listen for onplay, onpause events, timeupdate, volume ange, and so on, and then store them in records.

Canvas Canvas Change Monitor

canvas content changes do not throw events, so we can:

1. Collect canvas elements and update real-time content regularly
2. hack has some drawing API s to throw events

canvas monitoring research is not very in-depth, need further in-depth study

play

The idea is relatively simple, that is to get some information from the back end:

• Full Snapshot Virtual DOM
• Operation chain records
• Screen resolution
• doctype

With this information, you can first generate page DOM, including filtering script tags, and then create iframe, append to a container where you use a map to store DOM

function play(options = {}) {
  const { container, records = [], snapshot ={} } = options;
  const { vdom, doctype, clientHeight, clientWidth } = snapshot;
  this.nodeCache = {};
  this.records = records;
  this.container = container;
  this.snapshot = snapshot;
  this.iframe = document.createElement('iframe');
  const documentElement = createElement(vdom, (node) => {
    // Caching DOM
    const flowId = node.__flow && node.__flow.id;
    if (flowId) {
      this.nodeCache[flowId] = node;
    }
    // Filtering script
    return !(node.nodeType === Node.ELEMENT_NODE && node.tagName.toLowerCase() === 'script'); 
  });
    
  this.iframe.style.width = `${clientWidth}px`;
  this.iframe.style.height = `${clientHeight}px`;
  container.appendChild(iframe);
  const doc = iframe.contentDocument;
  this.iframeDocument = doc;
  doc.open();
  doc.write(`<!doctype ${doctype}><html><head></head><body></body></html>`);
  doc.close();
  doc.replaceChild(documentElement, doc.documentElement);
  this.execRecords();
}

function execRecords(preDuration = 0) {
  const record = this.records.shift();
  let node;
  if (record) {
    setTimeout(() => {
      switch (record.type) {
        // 'childList','characterData',
        // 'attributes','input','checked',
        // Handling of'focus','blur','play''pause'and other events
      }
      this.execRecords(record.duration);
    }, record.duration - preDuration)
  }
}

The above duration s are omitted from the above, which you can optimize according to your own play smoothness, see is more than one record as a frame or the original presentation.

On Fundebug

Fundebug Focus on JavaScript, WeChat applet, WeChat games, Alipay applet, React Native, Node.js and Java real-time application BUG monitoring. Since 2016, Fundebug has handled a total of 2 billion + errors. Payment customers include Sunshine Insurance, Walnut Programming, Lychee FM, Palm 1-to-1, Weimai, Youth League and many other brand enterprises. Welcome to all of you. Free trial!