More and more apps need to share mobile screens for others to watch, especially in the online education industry. Android has supported MediaProjection since 5.0. With MediaProjection, screen capture can be realized.
This library encapsulates the screen acquisition code. With a simple call, MediaProject permission application, H264 hard coding, error handling and other functions can be realized.
characteristic
- Adapt to Android higher version
- Asynchronous hard coding using MediaCodec
- Coding information configurable
- Notification bar display
- call chaining
use
ScreenShareKit.init(this)
.onH264{ buffer, isKeyFrame, ts ->
}.start()Github
realization
1 request user authorization screen acquisition
@TargetApi(Build.VERSION_CODES.M)
fun requestMediaProjection(encodeBuilder: EncodeBuilder){
this.encodeBuilder = encodeBuilder;
mediaProjectionManager = activity?.getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
startActivityForResult(mediaProjectionManager?.createScreenCaptureIntent(), 90000)
}startActivityForResult needs to be used in Activity or Fragment, and the authorization result will be recalled in onActivityResult. Therefore, we need to encapsulate this step so that it can get the result by callback to. Here we use an interface free Fragment, which is used by many libraries.
private val invisibleFragment : InvisibleFragment
get() {
val existedFragment = fragmentManager.findFragmentByTag(FRAGMENT_TAG)
return if (existedFragment != null) {
existedFragment as InvisibleFragment
} else {
val invisibleFragment = InvisibleFragment()
fragmentManager.beginTransaction()
.add(invisibleFragment, FRAGMENT_TAG)
.commitNowAllowingStateLoss()
invisibleFragment
}
}
fun start(){
invisibleFragment.requestMediaProjection(this)
}In this way, we can get the authorization result and MediaProjection object in onActivityResult in an interfaced Fragment.
2. Android 10
If targetSdkVersion is set at 29 and above, it will receive an exception after calling createVirtualDisplay after getting MediaProjection.
java.lang.SecurityException: Media projections require a foreground service of type ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION
This means that this operation needs to be performed in the foreground service.
Then we write a service and pass all the results obtained by onActivityResult.
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
intent?.let {
if(isStartCommand(it)){
val notification = NotificationUtils.getNotification(this)
startForeground(notification.first, notification.second) //Notification bar display
startProjection(
it.getIntExtra(RESULT_CODE, RESULT_CANCELED), it.getParcelableExtra(
DATA
)!!
)
}else if (isStopCommand(it)){
stopProjection()
stopSelf()
}
}
return super.onStartCommand(intent, flags, startId)
}In the startProjection method, we need to get the MediaProjectionManager, get the MediaProjection, and then create a virtual display.
private fun startProjection(resultCode: Int, data: Intent) {
val mpManager = getSystemService(MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
if (mMediaProjection == null) {
mMediaProjection = mpManager.getMediaProjection(resultCode, data)
if (mMediaProjection != null) {
mDensity = Resources.getSystem().displayMetrics.densityDpi
val windowManager = getSystemService(WINDOW_SERVICE) as WindowManager
mDisplay = windowManager.defaultDisplay
createVirtualDisplay()
mMediaProjection?.registerCallback(MediaProjectionStopCallback(), mHandler)
}
}
}
private fun createVirtualDisplay() {
mVirtualDisplay = mMediaProjection!!.createVirtualDisplay(
SCREENCAP_NAME,
encodeBuilder.encodeConfig.width,
encodeBuilder.encodeConfig.height,
mDensity,
DisplayManager.VIRTUAL_DISPLAY_FLAG_OWN_CONTENT_ONLY or DisplayManager.VIRTUAL_DISPLAY_FLAG_PUBLIC,
surface,
null,
mHandler
)
}In the createVirtualDisplay method, there is a Surface parameter. All actions on the screen will be mapped to this Surface. Here, we use MediaCodec to create an input Surface to receive the output of the screen and encode it.
3.MediaCodec code
private fun initMediaCodec() {
val format = MediaFormat.createVideoFormat(MIME, encodeBuilder.encodeConfig.width, encodeBuilder.encodeConfig.height)
format.apply {
setInteger(MediaFormat.KEY_COLOR_FORMAT,MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface) //Color format
setInteger(MediaFormat.KEY_BIT_RATE, encodeBuilder.encodeConfig.bitrate) //Code stream
setInteger(MediaFormat.KEY_BITRATE_MODE, MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR)
setInteger(MediaFormat.KEY_FRAME_RATE, encodeBuilder.encodeConfig.frameRate) //Number of frames
setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
}
codec = MediaCodec.createEncoderByType(MIME)
codec.apply {
setCallback(object : MediaCodec.Callback() {
override fun onInputBufferAvailable(codec: MediaCodec, index: Int) {
}
override fun onOutputBufferAvailable(
codec: MediaCodec,
index: Int,
info: MediaCodec.BufferInfo
) {
val outputBuffer:ByteBuffer?
try {
outputBuffer = codec.getOutputBuffer(index)
if (outputBuffer == null){
return
}
}catch (e:IllegalStateException){
return
}
val keyFrame = (info.flags and MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0
if (keyFrame){
configData = ByteBuffer.allocate(info.size)
configData.put(outputBuffer)
}else{
val data = createOutputBufferInfo(info,index,outputBuffer!!)
encodeBuilder.h264CallBack?.onH264(data.buffer,data.isKeyFrame,data.presentationTimestampUs)
}
codec.releaseOutputBuffer(index, false)
}
override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {
encodeBuilder.errorCallBack?.onError(ErrorInfo(-1,e.message.toString()))
}
override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) {
}
})
configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
surface = createInputSurface()
codec.start()
}
}Some conventional configurations are made above. MediaFormat can set some parameters for the encoder, such as code rate, frame rate, key frame interval, etc.
MediaCodec coding provides two modes: synchronous and asynchronous. Here, the callback is set asynchronously (asynchronous API 21 and above are available)
4. Encapsulation
In the onoutbufferavailable callback, I have called back the encoded data and determined whether it is a key frame or an ordinary frame. What's the use of encapsulating this library????
In fact, some third-party audio and video SDKs can be combined to directly stream the encoded screen stream data through the third-party SDK to realize the screen sharing function.
Here, take the pushExternalVideoFrame method of anyRTC audio and video SDK as an example
val rtcEngine = RtcEngine.create(this,"",RtcEvent())
rtcEngine.enableVideo()
rtcEngine.setExternalVideoSource(true,false,true)
rtcEngine.joinChannel("","111","","")
ScreenShareKit.init(this)
.onH264 {buffer, isKeyFrame, ts ->
rtcEngine.pushExternalVideoFrame(ARVideoFrame().apply {
val array = ByteArray(buffer.remaining())
buffer.get(array)
bufType = ARVideoFrame.BUFFER_TYPE_H264_EXTRA
timeStamp = ts
buf = array
height = Resources.getSystem().displayMetrics.heightPixels
stride = Resources.getSystem().displayMetrics.widthPixels
})
}.start()A few lines of code can realize screen acquisition, coding and transmission ~ very convenient
Pay attention to me and share knowledge every day~