Following Remember a blood case caused by passing a file handle once After that, the demo triggered another blood case, which is now recorded below.
This time I'm testing file handle transfer on linux, which does not have a STREAMS system.
The handle is therefore passed using the control message part in sendmsg/recvmsg of the unix domain socket.
The main modifications to the code are focused on sending and receiving fd s, which is how the code started and worked well.
1 #define MAXLINE 128 2 #define RIGHTSLEN CMSG_LEN(sizeof(int)) 3 #define CREDSLEN CMSG_LEN(sizeof(struct CREDSTRUCT)) 4 #define CONTROLLEN (RIGHTSLEN+CREDSLEN) 5 6 int send_fd (int fd, int fd_to_send) 7 { 8 struct iovec iov[1]; 9 struct msghdr msg; 10 struct cmsghdr *cmptr = NULL; 11 char buf[2]; 12 13 iov[0].iov_base = buf; 14 iov[0].iov_len = 2; 15 16 msg.msg_iov = iov; 17 msg.msg_iovlen = 1; 18 msg.msg_name = NULL; 19 msg.msg_namelen = 0; 20 msg.msg_flags = 0; 21 22 if (fd_to_send < 0) { 23 msg.msg_control = NULL; 24 msg.msg_controllen = 0; 25 buf[1] = -fd_to_send; 26 if (buf[1] == 0) 27 buf[1] = 1; 28 } else { 29 if ((cmptr = malloc(CONTROLLEN)) == NULL) { 30 fprintf (stderr, "malloc memory failed\n"); 31 return -1; 32 } 33 34 msg.msg_control = cmptr; 35 msg.msg_controllen = CONTROLLEN; 36 37 cmptr->cmsg_level = SOL_SOCKET; 38 cmptr->cmsg_type = SCM_RIGHTS; 39 cmptr->cmsg_len = CONTROLLEN; 40 41 *(int *) CMSG_DATA(cmptr) = fd_to_send; 42 buf[1] = 0; 43 } 44 45 buf[0] = 0; 46 if (sendmsg(fd, &msg, 0) != 2) { 47 free (cmptr); 48 return -1; 49 } 50 51 free (cmptr); 52 return 0; 53 }
The above is the Send Handle section, which focuses on lines 37-39 and sets the type of control message and the value of the handle.
The data message section in sendmsg that is used to accommodate scenarios where errors occur (an error code of -1~255 and a description can be provided), with the key information in the control section.
Here's how messages are received:
1 int recv_fd (int fd, uid_t *uidptr, ssize_t (*userfunc) (int, const void*, size_t)) 2 { 3 struct cmsghdr *cmptr = NULL; 4 int newfd, nr, status; 5 char *ptr; 6 char buf[MAXLINE]; 7 struct iovec iov[1]; 8 struct msghdr msg; 9 10 status = -1; 11 newfd = -1; 12 13 for (;;) { 14 iov[0].iov_base = buf; 15 iov[0].iov_len = sizeof (buf); 16 17 msg.msg_iov = iov; 18 msg.msg_iovlen = 1; 19 msg.msg_name = NULL; 20 msg.msg_namelen = 0; 21 22 if ((cmptr = malloc (CONTROLLEN)) == NULL) { 23 fprintf (stderr, "malloc error\n"); 24 return -1; 25 } 26 27 msg.msg_control = cmptr; 28 msg.msg_controllen = CONTROLLEN; 29 30 if ((nr = recvmsg (fd, &msg, 0)) < 0) { 31 fprintf (stderr, "recvmsg error\n"); 32 free (cmptr); 33 return -1; 34 } else if (nr == 0) { 35 fprintf (stderr, "connection closed by server\n"); 36 free (cmptr); 37 return -1; 38 } 39 40 for (ptr = buf; ptr < &buf[nr]; ) { 41 if (*ptr ++ == 0) { 42 if (ptr != &buf[nr-1]) { 43 fprintf (stderr, "message format error"); 44 free (cmptr); 45 return -1; 46 } 47 48 status = *ptr & 0xff; 49 if (status == 0) { 50 if (msg.msg_controllen != CONTROLLEN) { 51 fprintf (stderr, "status = 0 but no fd\n"); 52 free (cmptr); 53 return -1; 54 } 55 56 newfd = *(int *) CMSG_DATA(cmptr); 57 } else { 58 newfd = -status; 59 } 60 61 nr -= 2; 62 } 63 } 64 65 free(cmptr); 66 if (nr > 0 && (*userfunc)(STDERR_FILENO, buf, nr) != nr) 67 return -1; 68 69 if (status >= 0) 70 return newfd; 71 } 72 73 return -1; 74 }
The receive section focuses on line 56, where you get the file handle passed by the other party (note that this is not a simple value transfer!Refer to the previous article)
Other code handles error messages, and when an error occurs, calls userfunc to print the error message (the user typically passes a write).
In addition, the uidptr parameter in the interface is not useful, which is reserved for future extensions.
Compile and run using the previous demo (spipe_server.c / spipe_client.c). The output is as follows:
./spipe_server ./spipe_client create pipe 3.4 3 7 create temp file /tmp/outliqA3i with fd 4 seek to head send fd 4 to peer recv fd 3, position 0 create temp file /tmp/inaLr30i with fd 4 source: 3 7 seek to head send fd 4 recv fd 5 from peer, position 0 10
You can see that the value of the file handle passed in a new way has also changed (from 4 to 3), and the file offset needs to be reset, or it will fall into the hole mentioned in the previous article.
The problem arises when adding some code to pass the sending process credentials (such as uid), where the sender needs to pass two control submessages (representing the handle and the credentials, respectively) and the receiver needs to process two.
The new send code is as follows:
1 #define MAXLINE 128 2 #if defined(SCM_CREDS) // on BSD 3 #define CREDSTRUCT cmsgcred 4 #define CR_UID cmcred_uid 5 #define CREDOPT LOCAL_PEERCRED 6 #define SCM_CREDTYPE SCM_CREDS 7 #elif defined(SCM_CREDENTIALS) // on linux 8 #define CREDSTRUCT ucred 9 #define CR_UID uid 10 #define CREDOPT SO_PASSCRED 11 #define SCM_CREDTYPE SCM_CREDENTIALS 12 #else 13 #error passing credentials is unsupported! 14 #endif 15 16 #define RIGHTSLEN CMSG_LEN(sizeof(int)) 17 #define CREDSLEN CMSG_LEN(sizeof(struct CREDSTRUCT)) 18 #define CONTROLLEN (RIGHTSLEN+CREDSLEN) 19 20 21 int send_fd (int fd, int fd_to_send) 22 { 23 struct iovec iov[1]; 24 struct msghdr msg; 25 struct cmsghdr *cmptr = NULL; 26 char buf[2]; 27 struct CREDSTRUCT *credp; 28 struct cmsghdr *cmp; 29 30 iov[0].iov_base = buf; 31 iov[0].iov_len = 2; 32 33 msg.msg_iov = iov; 34 msg.msg_iovlen = 1; 35 msg.msg_name = NULL; 36 msg.msg_namelen = 0; 37 msg.msg_flags = 0; 38 39 if (fd_to_send < 0) { 40 msg.msg_control = NULL; 41 msg.msg_controllen = 0; 42 buf[1] = -fd_to_send; 43 if (buf[1] == 0) 44 buf[1] = 1; 45 } else { 46 if ((cmptr = malloc(CONTROLLEN)) == NULL) { 47 fprintf (stderr, "malloc memory failed\n"); 48 return -1; 49 } 50 51 msg.msg_control = cmptr; 52 msg.msg_controllen = CONTROLLEN; 53 54 cmp = cmptr; 55 cmp->cmsg_level = SOL_SOCKET; 56 cmp->cmsg_type = SCM_RIGHTS; 57 cmp->cmsg_len = RIGHTSLEN; 58 *(int *) CMSG_DATA(cmp) = fd_to_send; 59 60 cmp = CMSG_NXTHDR(&msg, cmp); 61 cmp->cmsg_level = SOL_SOCKET; 62 cmp->cmsg_type = SCM_CREDTYPE; 63 cmp->cmsg_len = CREDSLEN; 64 credp = (struct CREDSTRUCT *) CMSG_DATA(cmp); 65 66 # if defined(SCM_CREDENTIALS) 67 // only linux need to set members of this struct ! 68 credp->uid = getuid (); 69 credp->gid = getegid (); 70 credp->pid = getpid (); 71 # endif 72 buf[1] = 0; 73 } 74 75 buf[0] = 0; 76 if (sendmsg(fd, &msg, 0) != 2) { 77 free (cmptr); 78 return -1; 79 } 80 81 free (cmptr); 82 return 0; 83 }
The first macro definitions were used to distinguish details on linux from bsd, with emphasis on lines 55-64, where handles and credentials were set.
The size of the control message CONTROLLEN is then accumulated by the length of the two-part message (RIGHTSLEN and CREDLEN), and the allocated memory is as large.
Look again at the reception section:
1 int recv_fd (int fd, uid_t *uidptr, ssize_t (*userfunc) (int, const void*, size_t)) 2 { 3 struct cmsghdr *cmptr = NULL; 4 5 int newfd, nr, status; 6 char *ptr; 7 char buf[MAXLINE]; 8 struct iovec iov[1]; 9 struct msghdr msg; 10 11 status = -1; 12 newfd = -1; 13 14 const int on = -1; 15 struct cmsghdr *cmp; 16 struct CREDSTRUCT *credp; 17 if (setsockopt (fd, SOL_SOCKET, CREDOPT, &on, sizeof(int)) < 0) { 18 fprintf (stderr, "setsockopt for %d failed\n", CREDOPT); 19 return -1; 20 } 21 22 for (;;) { 23 iov[0].iov_base = buf; 24 iov[0].iov_len = sizeof (buf); 25 26 msg.msg_iov = iov; 27 msg.msg_iovlen = 1; 28 msg.msg_name = NULL; 29 msg.msg_namelen = 0; 30 31 if ((cmptr = malloc (CONTROLLEN)) == NULL) { 32 fprintf (stderr, "malloc error\n"); 33 return -1; 34 } 35 36 msg.msg_control = cmptr; 37 msg.msg_controllen = CONTROLLEN; 38 39 if ((nr = recvmsg (fd, &msg, 0)) < 0) { 40 fprintf (stderr, "recvmsg error\n"); 41 free (cmptr); 42 return -1; 43 } else if (nr == 0) { 44 fprintf (stderr, "connection closed by server\n"); 45 free (cmptr); 46 return -1; 47 } 48 49 for (ptr = buf; ptr < &buf[nr]; ) { 50 if (*ptr ++ == 0) { 51 if (ptr != &buf[nr-1]) { 52 fprintf (stderr, "message format error"); 53 free (cmptr); 54 return -1; 55 } 56 57 status = *ptr & 0xff; 58 if (status == 0) { 59 if (msg.msg_controllen != CONTROLLEN) { 60 fprintf (stderr, "status = 0 but no fd\n"); 61 free (cmptr); 62 return -1; 63 } 64 65 for (cmp = CMSG_FIRSTHDR(&msg); cmp != NULL; cmp = CMSG_NXTHDR(&msg, cmp)) { 66 if (cmp->cmsg_level != SOL_SOCKET) { 67 fprintf (stderr, "ignore unknown socket level %d\n", cmp->cmsg_level); 68 continue; 69 } 70 71 switch (cmp->cmsg_type) { 72 case SCM_RIGHTS: 73 newfd = *(int *) CMSG_DATA(cmp); 74 break; 75 case SCM_CREDTYPE: 76 credp = (struct CREDSTRUCT *) CMSG_DATA(cmp); 77 *uidptr = credp->CR_UID; 78 break; 79 default: 80 fprintf (stderr, "ignore unknown msg type %d\n", cmp->cmsg_type); 81 break; 82 } 83 } 84 } else { 85 newfd = -status; 86 } 87 88 nr -= 2; 89 } 90 } 91 92 free(cmptr); 93 if (nr > 0 && (*userfunc)(STDERR_FILENO, buf, nr) != nr) 94 return -1; 95 96 if (status >= 0) 97 return newfd; 98 } 99 100 return -1; 101 }
The focus is divided into two parts:
14-20 lines, set unix domain socket to receive credential information;
Lines 65-83 read the handle and credential information in the control message, respectively. Here we take the uid information of the sending process as credentials and return it to the upper caller.
Similar to sending a message, CMSG_FIRSTHDR and CMSG_NXTHDR provided by the system are used here to traverse subparts in the control message.
Re-compile and run demo, but find an error:
./spipe_server ./spipe_client create pipe 3.4 3 7 create temp file /tmp/outgQY1Y4 with fd 4 seek to head send fd 4 to peer recv fd 3, uid 500, position 0 create temp file /tmp/invVgKW4 with fd 4 source: 3 7 seek to head connection closed by server recv fd from peer failed, error -1
From the output log, the handle and credentials sent from the server to the client for the first time are OK (line 7), and then the client makes a mistake when it finishes processing the message return.
Firstly, locate the error code, return it here (send_fd) from the client, and add some logs:
1 if ((cmptr = malloc(CONTROLLEN)) == NULL) { 2 fprintf (stderr, "malloc memory failed\n"); 3 return -1; 4 } 5 6 msg.msg_control = cmptr; 7 msg.msg_controllen = CONTROLLEN; 8 9 cmp = cmptr; 10 cmp->cmsg_level = SOL_SOCKET; 11 cmp->cmsg_type = SCM_RIGHTS; 12 cmp->cmsg_len = RIGHTSLEN; 13 *(int *) CMSG_DATA(cmp) = fd_to_send; 14 fprintf (stderr, "add fd with len %d\n", RIGHTSLEN); 15 16 cmp = CMSG_NXTHDR(&msg, cmp); 17 cmp->cmsg_level = SOL_SOCKET; 18 cmp->cmsg_type = SCM_CREDTYPE; 19 cmp->cmsg_len = CREDSLEN; 20 credp = (struct CREDSTRUCT *) CMSG_DATA(cmp); 21 fprintf (stderr, "add credential with len %d\n", CREDSLEN); 22 23 # if defined(SCM_CREDENTIALS) 24 // only linux need to set members of this struct ! 25 credp->uid = getuid (); 26 credp->gid = getegid (); 27 credp->pid = getpid (); 28 fprintf (stderr, "set uid %d, gid %d, pid %d\n", credp->uid, credp->gid, credp->pid); 29 # endif 30 buf[1] = 0;
Marked yellow is the newly added output log, compile and run again:
./spipe_server ./spipe_client create pipe 3.4 3 7 create temp file /tmp/outivt2Og with fd 4 seek to head add fd with len 16 add credential with len 24 set uid 500, gid 500, pid 12071 send fd 4 to peer recv fd 3, uid 500, position 0 create temp file /tmp/inHqRwMg with fd 4 source: 3 7 seek to head add fd with len 16 connection closed by server recv fd from peer failed, error -1
You can see that all three logs were output correctly the first time they were delivered, and only the first log was output when they were returned.
So obviously there was a problem with the code between the first log and the second log.Looking left and right, I can't see what's wrong with this piece. Can CMSG_NXTHDR provided by the system go wrong?
There are two more logs on this side:
1 if ((cmptr = malloc(CONTROLLEN)) == NULL) { 2 fprintf (stderr, "malloc memory failed\n"); 3 return -1; 4 } 5 6 msg.msg_control = cmptr; 7 msg.msg_controllen = CONTROLLEN; 8 9 cmp = cmptr; 10 cmp->cmsg_level = SOL_SOCKET; 11 cmp->cmsg_type = SCM_RIGHTS; 12 cmp->cmsg_len = RIGHTSLEN; 13 *(int *) CMSG_DATA(cmp) = fd_to_send; 14 fprintf (stderr, "add fd with len %d\n", RIGHTSLEN); 15 fprintf (stderr, "cmsghdr = %d, cmsglen = %d, after align = %d, control len = %d\n", sizeof(struct cmsghdr), CREDSLEN, CMSG_ALIGN(CREDSLEN), CONTROLLEN); 16 17 cmp = CMSG_NXTHDR(&msg, cmp); 18 fprintf (stderr, "cmp = %p\n", cmp); 19 cmp->cmsg_level = SOL_SOCKET; 20 cmp->cmsg_type = SCM_CREDTYPE; 21 cmp->cmsg_len = CREDSLEN; 22 credp = (struct CREDSTRUCT *) CMSG_DATA(cmp); 23 fprintf (stderr, "add credential with len %d\n", CREDSLEN); 24 25 # if defined(SCM_CREDENTIALS) 26 // only linux need to set members of this struct ! 27 credp->uid = getuid (); 28 credp->gid = getegid (); 29 credp->pid = getpid (); 30 fprintf (stderr, "set uid %d, gid %d, pid %d\n", credp->uid, credp->gid, credp->pid); 31 # endif 32 buf[1] = 0;
The second log is the main suspect, to see if the pointer is empty; the first is to suspect that the block size calculation is incorrect, resulting in insufficient allocated memory and range errors when the pointer increases, so print various lengths here to verify.
After running again, there are some more outputs:
./spipe_server ./spipe_client create pipe 3.4 3 7 create temp file /tmp/out7UgSYZ with fd 4 seek to head add fd with len 16 cmsghdr = 12, cmsglen = 24, after align = 24, control len = 40 cmp = 0x9ded018 add credential with len 24 set uid 500, gid 500, pid 12100 send fd 4 to peer recv fd 3, uid 500, position 0 create temp file /tmp/inC3nyWZ with fd 4 source: 3 7 seek to head add fd with len 16 cmsghdr = 12, cmsglen = 24, after align = 24, control len = 40 cmp = (nil) connection closed by server recv fd from peer failed, error -1
A magical place appears, the same code, the same size, the first pointer is normal; the second is empty!
The crash point was found, but it was still confused. It looks like the data blocks are aligned and the calculations are okay. Is there something wrong with the macro (CMSG_NXTHDR) provided by this system?
Look through the header file to find the definition of this section (my system, at /usr/include/bits/socket.h (L311)):
1 __EXTERN_INLINE struct cmsghdr * 2 __NTH (__cmsg_nxthdr (struct msghdr *__mhdr, struct cmsghdr *__cmsg)) 3 { 4 if ((size_t) __cmsg->cmsg_len < sizeof (struct cmsghdr)) 5 /* The kernel header does this so there may be a reason. */ 6 return 0; 7 8 __cmsg = (struct cmsghdr *) ((unsigned char *) __cmsg 9 + CMSG_ALIGN (__cmsg->cmsg_len)); 10 if ((unsigned char *) (__cmsg + 1) > ((unsigned char *) __mhdr->msg_control 11 + __mhdr->msg_controllen) 12 || ((unsigned char *) __cmsg + CMSG_ALIGN (__cmsg->cmsg_len) 13 > ((unsigned char *) __mhdr->msg_control + __mhdr->msg_controllen))) 14 /* No more entries. */ 15 return 0; 16 return __cmsg; 17 }
This INLINE function mainly contains three judgments.
1) Submessage length is less than header length, returning null;
2) The header of the next sub-message exceeds the end of the message and returns null;
3) The message body of the next sub-message exceeds the end of the message and returns null;
It is inconvenient to modify the system code directly. Copy this function locally and rename it my_cmsg_nxthdr. Add the log output under each judgment:
1 struct cmsghdr *my_cmsg_nxthdr (struct msghdr *__mhdr, struct cmsghdr *__cmsg) 2 { 3 if ((size_t) __cmsg->cmsg_len < sizeof (struct cmsghdr)) { 4 /* The kernel header does this so there may be a reason. */ 5 fprintf (stderr, "in step1\n"); 6 return 0; 7 } 8 9 fprintf (stderr, "%p: cmsg_len %u, cmsg_level %d, cmsg_type %d\n", __cmsg, __cmsg->cmsg_len, __cmsg->cmsg_level, __cmsg->cmsg_type); 10 __cmsg = (struct cmsghdr *) ((unsigned char *) __cmsg + CMSG_ALIGN (__cmsg->cmsg_len)); 11 if ((unsigned char *) (__cmsg + 1) > ((unsigned char *) __mhdr->msg_control + __mhdr->msg_controllen)) { 12 fprintf (stderr, "in step2\n"); 13 return 0; 14 } 15 16 fprintf (stderr, "%p: cmsg_len %u, cmsg_level %d, cmsg_type %d\n", __cmsg, __cmsg->cmsg_len, __cmsg->cmsg_level, __cmsg->cmsg_type); 17 if (((unsigned char *) __cmsg + CMSG_ALIGN (__cmsg->cmsg_len) > ((unsigned char *) __mhdr->msg_control + __mhdr->msg_controllen))) { 18 /* No more entries. */ 19 fprintf (stderr, "in step3\n"); 20 fprintf (stderr, "msg len %d, after align %d, msg control %d\n", __cmsg->cmsg_len, CMSG_ALIGN(__cmsg->cmsg_len), __mhdr->msg_controllen); 21 return 0; 22 } 23 24 fprintf (stderr, "in final step\n"); 25 return __cmsg; 26 }
To facilitate the output of logs based on different criteria, the criteria are split here.
Run demo again with the following output:
./spipe_server ./spipe_client create pipe 3.4 3 7 create temp file /tmp/outh7NhIs with fd 4 seek to head add fd with len 16 cmsghdr = 12, cmsglen = 24, after align = 24, control len = 40 0x9336008: cmsg_len 16, cmsg_level 1, cmsg_type 1 0x9336018: cmsg_len 0, cmsg_level 0, cmsg_type 0 in final step cmp = 0x9336018 add credential with len 24 set uid 500, gid 500, pid 12171 send fd 4 to peer recv fd 3, uid 500, position 0 create temp file /tmp/inoJMmKs with fd 4 source: 3 7 seek to head add fd with len 16 cmsghdr = 12, cmsglen = 24, after align = 24, control len = 40 0x904d008: cmsg_len 16, cmsg_level 1, cmsg_type 1 0x904d018: cmsg_len 500, cmsg_level 500, cmsg_type 16 in step3 msg len 500, after align 500, msg control 40 cmp = (nil) connection closed by server recv fd from peer failed, error -1
It was the third judgment that had a problem (Line 24)!
The total message length is 16 + 24 = 40, whereas the second sub-message here has a single length of 500, which is clearly out of bounds.
But the length of the second sub-message is clearly 24. Where did it run out of 500?
And its other fields are obviously incorrect, such as message level 500 and message type 16!
Preliminary confirmation is that this memory is messed up, and from the message pointer printed earlier (0x904d008 and 0x904d018), the allocated size is OK, so the memory boundary problem is eliminated first;
And then what we set up...Wait...We don't seem to have set the content of the second sub-message yet!!
......
Spam data!!
......
No empty garbage data after malloc!!
......
This is also the reason why the first call was okay and the second time it fell into a pit, there is some randomness as system memory is allocated and recycled!
Once you find the reason, the modification is simple, replacing malloc with calloc, or simply adding a memset to empty memory:
1 if ((cmptr = malloc(CONTROLLEN)) == NULL) { 2 fprintf (stderr, "malloc memory failed\n"); 3 return -1; 4 } 5 6 // important on linux, garbage data may mess cmsg_len fields, 7 // and cause CMSG_NXTHDR return null on protection. 8 memset (cmptr, 0, CONTROLLEN); 9 msg.msg_control = cmptr; 10 msg.msg_controllen = CONTROLLEN;
Run demo again, everything is OK:
./spipe_server ./spipe_client create pipe 3.4 3 7 create temp file /tmp/outqsTYkp with fd 4 seek to head add fd with len 16 cmsghdr = 12, cmsglen = 24, after align = 24, control len = 40 0x814c008: cmsg_len 16, cmsg_level 1, cmsg_type 1 0x814c018: cmsg_len 0, cmsg_level 0, cmsg_type 0 in final step cmp = 0x814c018 add credential with len 24 set uid 500, gid 500, pid 12207 send fd 4 to peer recv fd 3, uid 500, position 0 create temp file /tmp/in3ntkip with fd 4 source: 3 7 seek to head add fd with len 16 cmsghdr = 12, cmsglen = 24, after align = 24, control len = 40 0x8389008: cmsg_len 16, cmsg_level 1, cmsg_type 1 0x8389018: cmsg_len 0, cmsg_level 0, cmsg_type 0 in final step cmp = 0x8389018 add credential with len 24 set uid 500, gid 500, pid 12208 send fd 4 recv fd 5, uid 500 from peer, position 0 10
This debug finds a flaw in the classic APU example (randomness is great, the Master just didn't encounter it, maybe your machine is no longer there).
But looking back at this scenario, we can't count all on coder. I feel that the CMSG_NXTHDR macro provided by the system is also problematic:
If I have not set the next sub-message before calling this one, can't I use it? Too many checks just make me feel awkward. In one word: bad reviews!Haha~