


remctl Protocol                                               R. Allbery
                                                     Stanford University
                                                            October 2011


              remctl: Remote Authenticated Command Service

Abstract

   This document specifies the remctl wire protocol, used to send
   commands and arguments to a remote system and receive the results of
   executing that command.  The protocol uses GSS-API and Kerberos v5
   for authentication, confidentiality, and integrity protection.  Both
   the current (version 3) protocol and the older version 1 protocol are
   described.  The version 1 protocol should only be implemented for
   backward compatibility.



































Allbery                                                         [Page 1]

                                 remctl                     October 2011


Table of Contents

   1.  Basic Packet Format  . . . . . . . . . . . . . . . . . . . . .  3
   2.  Network Protocol (version 3) . . . . . . . . . . . . . . . . .  4
     2.1.  Session Sequence . . . . . . . . . . . . . . . . . . . . .  4
     2.2.  Message Format . . . . . . . . . . . . . . . . . . . . . .  5
     2.3.  Protocol Version Negotiation . . . . . . . . . . . . . . .  5
     2.4.  MESSAGE_COMMAND  . . . . . . . . . . . . . . . . . . . . .  6
     2.5.  MESSAGE_OUTPUT and MESSAGE_STATUS  . . . . . . . . . . . .  7
     2.6.  MESSAGE_ERROR  . . . . . . . . . . . . . . . . . . . . . .  8
     2.7.  MESSAGE_QUIT . . . . . . . . . . . . . . . . . . . . . . .  9
     2.8.  MESSAGE_NOOP . . . . . . . . . . . . . . . . . . . . . . .  9
   3.  Network Protocol (version 1) . . . . . . . . . . . . . . . . . 10
   4.  Security Considerations  . . . . . . . . . . . . . . . . . . . 12
   Appendix A.  Acknowledgements  . . . . . . . . . . . . . . . . . . 13
   Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 14



































Allbery                                                         [Page 2]

                                 remctl                     October 2011


1.  Basic Packet Format

   The remctl network protocol consists of data packets sent from a
   client to a server or a server to a client over a TCP connection.
   The remctl protocol may be used over any port, but the IANA-
   registered port and the RECOMMENDED default for the protocol is 4373.
   Each data packet has the following format:

       1 octet     flags
       4 octets    length
       <length>    data payload

   The total size of each token, including the five octet prefix, MUST
   NOT be larger than 1,048,576 octets (1MB).

   The flag octet contains one or more of the following values, combined
   with binary xor:

       0x01        TOKEN_NOOP
       0x02        TOKEN_CONTEXT
       0x04        TOKEN_DATA
       0x08        TOKEN_MIC
       0x10        TOKEN_CONTEXT_NEXT
       0x20        TOKEN_SEND_MIC
       0x40        TOKEN_PROTOCOL

   Only TOKEN_CONTEXT, TOKEN_CONTEXT_NEXT, TOKEN_DATA, and
   TOKEN_PROTOCOL are used for packets for version 2 and of the
   protocol.  The other flags are used only with the legacy version 1
   protocol.

   The length field is a four-octet length in network byte order,
   specifying the number of octets in the following data payload.

   The data payload is empty, the results of gss_accept_sec_context, the
   results of gss_init_sec_context, or a data payload protected with
   gss_wrap.  The length of the data passed to gss_wrap MUST NOT be
   larger than 65,536 octets (64KB), even if the underlying Kerberos
   implementation supports longer input buffers.












Allbery                                                         [Page 3]

                                 remctl                     October 2011


2.  Network Protocol (version 3)

2.1.  Session Sequence

   A remctl connection is always initiated by a client opening a TCP
   connection to a server.  The protocol then proceeds as follows:

   1.  Client sends message with an empty payload and flags TOKEN_NOOP,
       TOKEN_CONTEXT_NEXT, and TOKEN_PROTOCOL (0x51).  If the client
       doesn't include TOKEN_PROTOCOL, it is speaking the version 1
       protocol, and the server MUST either drop the connection or fall
       back to the version 1 protocol.  This initial message is useless
       in a pure version 2 protocol world and is done only for backward
       compatibility with the version 1 protocol.

   2.  Client calls gss_init_sec_context and replies with the results
       and TOKEN_CONTEXT and TOKEN_PROTOCOL (0x42).  The client MUST
       pass GSS_C_MUTUAL_FLAG, GSS_C_CONF_FLAG, and GSS_C_INTEG_FLAG as
       requested flags and SHOULD pass GSS_C_REPLAY_FLAG and
       GSS_C_SEQUENCE_FLAG.

   3.  Server replies with the results of gss_accept_sec_context and
       flags TOKEN_CONTEXT and TOKEN_PROTOCOL (0x42).  If the server
       doesn't include TOKEN_PROTOCOL in the flags, it is speaking the
       version 1 protocol, and the client MUST either drop the
       connection or fall back to the version 1 protocol.

   4.  Client passes data to gss_init_sec_context and replies with the
       results and TOKEN_CONTEXT and TOKEN_PROTOCOL (0x42).  The client
       must pass GSS_C_MUTUAL_FLAG, GSS_C_CONF_FLAG, and
       GSS_C_INTEG_FLAG as requested flags and SHOULD pass
       GSS_C_REPLAY_FLAG and GSS_C_SEQUENCE_FLAG.

   5.  Server and client repeat, passing in the payload from the last
       packet from the other side, for as long as GSS-API indicates that
       continuation is required.  If either side drops TOKEN_PROTOCOL
       from the flags, it is an considered an error and the connect MUST
       be dropped.  (This could be a down-negotiation attack.)  After
       the establishment of the security context, both client and server
       MUST confirm that GSS_C_MUTUAL_FLAG, GSS_C_CONF_FLAG, and
       GSS_C_INTEG_FLAG are set in the resulting security context and
       MUST immediately close the connection if this is not the case.

   6.  After the security context has been established, the client and
       server exchange commands and responses as described below.  All
       commands are sent with flags TOKEN_DATA and TOKEN_PROTOCOL (0x44)
       and the data payload of all packets is protected with gss_wrap.
       The conf_req_flag parameter of gss_wrap MUST be set to non-zero,



Allbery                                                         [Page 4]

                                 remctl                     October 2011


       requesting both confidentiality and integrity services.

2.2.  Message Format

   All client and server messages will use the following format inside
   the data payload.  This is the format of the message before passing
   it to gss_wrap for confidentiality and integrity protection.

       1 octet     protocol version
       1 octet     message type
       <command-specific data>

   The protocol version sent for all messages should be 2 with the
   exception of MESSAGE_NOOP, which should have a protocol version of 3.
   The version 1 protocol does not use this message format, and
   therefore a protocol version of 1 is invalid.  See below for protocol
   version negotiation.

   The message type is one of the following constants:

       1   MESSAGE_COMMAND
       2   MESSAGE_QUIT
       3   MESSAGE_OUTPUT
       4   MESSAGE_STATUS
       5   MESSAGE_ERROR
       6   MESSAGE_VERSION
       7   MESSAGE_NOOP

   The first two message types are client messages and MUST NOT be sent
   by the server.  The remaining message types are server messages and
   MUST NOT by sent by the client.

   All of these message types were introduced in protocol version 2
   except for MESSAGE_NOOP, which is a protocol version 3 message.

2.3.  Protocol Version Negotiation

   If the server ever receives a message from a client that claims a
   protocol version higher than the server supports, the server MUST
   otherwise ignore the contents of the message and SHOULD respond with
   a message type of MESSAGE_VERSION and the following message payload:

       1 octet     highest supported version

   The client MUST then either send only messages supported at that
   protocol version or lower or send MESSAGE_QUIT and close the
   connection.




Allbery                                                         [Page 5]

                                 remctl                     October 2011


   Currently, there are only two meaningful values for the highest
   supported version: 3, which indicates everything in this
   specification is supported, or 2, which indicates that everything
   except MESSAGE_NOOP is supported.

2.4.  MESSAGE_COMMAND

   Most client messages will be of type MESSAGE_COMMAND, which has the
   following format:

       1 octet     keep-alive flag
       1 octet     continue status
       4 octets    number of arguments
       4 octets    argument length
       <length>    argument
       ...

   If the keep-alive flag is 0, the server SHOULD close the connection
   after processing the command.  If it is 1, the server SHOULD leave
   the connection open (up to a timeout period) and wait for more
   commands.  This is similar to HTTP keep-alive.

   If the continue status is 1, it indicates that there is more data
   coming.  The server should accept the data sent, buffer or process
   it, and wait for additional messages before responding.  If the the
   continue status is 2, it indicates that this message is logically a
   part of the previous message (which MUST have had a continue status
   of 1 or 2) and still has more data coming.  If the continue status is
   3, it says that this message is logically part of the previous
   message, like 2, but it also says that this is the end of the
   command.

   A continuation of a message starts with the keep-alive flag and
   continue status and then the next chunk of data.  To reconstruct a
   continued message, remove the first two octets from each chunk and
   concatenate the pieces together.  The result is the portion of a
   MESSAGE_COMMAND starting with the number of arguments.  Messages may
   be broken into multiple MESSAGE_COMMANDs even in the middle of the
   number of arguments or an argument length.  In other words, the first
   three octets of the number of arguments could be in the first
   MESSAGE_COMMAND (with continue status 1) and the last octet would
   then be in the next MESSAGE_COMMAND (with continue status 2 or 3).

   For as long as the continue status is 1 or 2, the next message from
   the client MUST be either another MESSAGE_COMMAND with a continue
   status of 2 or 3 or a MESSAGE_QUIT.  In other words, other message
   types MUST NOT be intermixed with continued commands, but
   MESSAGE_QUIT may be sent by the client in the middle of a continued



Allbery                                                         [Page 6]

                                 remctl                     October 2011


   command to abort that command.  If the server receives MESSAGE_QUIT
   from the client before receiving a MESSAGE_COMMAND with a status of 3
   (indicating the end of the command), the command MUST be discarded
   and not executed.

   If a client sends an invalid sequence of MESSAGE_COMMAND messages
   that violate the continuation rules described above, the server
   SHOULD reply with a MESSAGE_ERROR message, generally with either
   ERROR_BAD_TOKEN, ERROR_UNKNOWN_MESSAGE, ERROR_BAD_COMMAND, or
   ERROR_UNEXPECTED_MESSAGE error codes.  It MUST discard the partial
   command without acting on it.  The client cannot correct an error in
   a continued MESSAGE_COMMAND stream by resending the previous part.
   It MUST start again at the beginning with a MESSAGE_COMMAND with a
   continue status of 0 or 1.

   Number of arguments is a four-octet number in network byte order that
   gives the total number of command arguments.  For each argument,
   there is then a length and argument data pair, where the length is a
   four-octet number in network byte order indicating the number of
   octets of data in the following argument.  Argument length may be 0.
   Commands with no arguments are permitted by the protocol.

   Servers may impose limits on the number of arguments and the size of
   argument data to limit resource usage.  If the client message exceeds
   one of those limits, the server MUST respond with MESSAGE_ERROR with
   an error code of ERROR_TOOMANY_ARGS or ERROR_TOOMUCH_DATA as
   appropriate.

2.5.  MESSAGE_OUTPUT and MESSAGE_STATUS

   The server response to MESSAGE_COMMAND is zero or more MESSAGE_OUTPUT
   messages followed by either a MESSAGE_STATUS or a MESSAGE_ERROR
   response.  Each MESSAGE_OUTPUT message has the following format:

       1 octet     output stream
       4 octets    output length
       <length>    output

   The output stream is either 1 for standard output or 2 for standard
   error.  Output length is a four-octet number in network byte order
   that specifies the length of the following output data.

   The MESSAGE_STATUS message has the following format:

       1 octet     exit status

   MESSAGE_STATUS indicates the command has finished and returns the
   final exit stauts of the command.  Exit status is 0 for success and



Allbery                                                         [Page 7]

                                 remctl                     October 2011


   non-zero for failure, where the meaning of non-zero exit statuses is
   left to the application to define.  (This is identical to a Unix
   command exit status.)

   Unless the MESSAGE_COMMAND message from the client had the keep-alive
   flag set to 1, the server MUST close the network connection
   immediately after sending the MESSAGE_STATUS response message.

2.6.  MESSAGE_ERROR

   At any point before sending MESSAGE_STATUS, the server may respond
   with MESSAGE_ERROR if some error occurred.  This can be the first
   response after a MESSAGE_COMMAND, or it may be sent after one or more
   MESSAGE_OUTPUT messages.  The format of MESSAGE_ERROR is as follows:

       4 octets    error code
       4 octets    message length
       <length>    error message

   The error code is a four-octet number in network byte order
   indicating the type of error.  The error code may be one of the
   following values:

       1  ERROR_INTERNAL           Internal server failure
       2  ERROR_BAD_TOKEN          Invalid format in token
       3  ERROR_UNKNOWN_MESSAGE    Unknown message type
       4  ERROR_BAD_COMMAND        Invalid command format in token
       5  ERROR_UNKNOWN_COMMAND    Unknown command
       6  ERROR_ACCESS             Access denied
       7  ERROR_TOOMANY_ARGS       Argument count exceeds server limit
       8  ERROR_TOOMUCH_DATA       Argument size exceeds server limit
       9  ERROR_UNEXPECTED_MESSAGE Message type not valid now

   Additional error codes may be added without changing the version of
   the remctl protocol, so clients MUST accept error codes other than
   the ones above.

   The message length is a four-octet number in network byte order that
   specifies the length in octets of the following error message.  The
   error message is a free-form informational message intended for human
   consumption and MUST NOT be interpreted by an automated process.
   Software should instead use the error code.

   Unless the MESSAGE_COMMAND message from the client had the keep-alive
   flag set to 1, the server MUST close the network connection
   immediately after sending the MESSAGE_ERROR response message.
   Otherwise, the server SHOULD still honor that flag, although the
   server MAY terminate the connection after an unreasonable number of



Allbery                                                         [Page 8]

                                 remctl                     October 2011


   errors.

2.7.  MESSAGE_QUIT

   MESSAGE_QUIT is a way of terminating the connection cleanly if the
   client asked for keep-alive and then decided not to use it.  There is
   no message body.  Upon receiving this message, the server MUST
   immediately close the connection.

2.8.  MESSAGE_NOOP

   MESSAGE_NOOP provides a way for a client to keep the connection open
   to a remctl server, including through firewall session timeouts and
   similar network constraints that require periodic activity, without
   sending new commands.  There is no body.  When the client sends a
   MESSAGE_NOOP message, the server replies with a MESSAGE_NOOP message.

   Note that MESSAGE_NOOP was introduced in protocol version 3 and
   therefore should be marked accordingly.  Clients should be prepared
   for older servers to reply with MESSAGE_VERSION instead of
   MESSAGE_NOOP.






























Allbery                                                         [Page 9]

                                 remctl                     October 2011


3.  Network Protocol (version 1)

   The old network protocol supported only 64KB of data payload, only a
   single command and response, and had some additional unnecessary
   protocol components.  It SHOULD NOT be used by clients, but MAY be
   supported by servers for backward compatibility.  It is recognized by
   the server and client by the lack of TOKEN_PROTOCOL in the flags of
   the initial security context negotiation.

   The old protocol always uses the following steps:

   1.   Client opens TCP connection to server.

   2.   Client sends message with flags TOKEN_NOOP and
        TOKEN_CONTEXT_NEXT and an empty payload.

   3.   Client calls gss_init_sec_context and sends message with the
        results and flags TOKEN_CONTEXT.  The client MUST pass
        GSS_C_MUTUAL_FLAG, GSS_C_CONF_FLAG, and GSS_C_INTEG_FLAG as
        requested flags and SHOULD pass GSS_C_REPLAY_FLAG and
        GSS_C_SEQUENCE_FLAG, although the version one protocol does not
        check the results of this negotiation.

   4.   Server replies with the results of gss_accept_sec_context and
        flags TOKEN_CONTEXT.

   5.   Client calls gss_init_sec_context again with the data from the
        server and replies with the results and flags TOKEN_CONTEXT,
        using the same requested flags as described above.

   6.   Server and client repeat, passing in the payload from the last
        packet from the other side, for as long as GSS-API indicates
        that continuation is required.  Each of these packets have only
        TOKEN_CONTEXT set in the flags.

   7.   Client sends command with flags TOKEN_DATA and TOKEN_SEND_MIC
        and the following payload format: four-octet number of
        arguments, and then for each argument, a four-octet length and
        then the argument value.  All numbers are in network type order.
        The payload MUST be protected with gss_wrap and the
        conf_req_flag parameter of gss_wrap MUST be set to non-zero,
        requesting both confidentiality and integrity services.

   8.   Server accepts and decrypts data, generates a MIC with
        gss_get_mic, and sends the MIC back to the client with flags
        TOKEN_MIC.  This is the only packet that isn't encrypted with
        gss_wrap.  Client receives and then SHOULD verify this MIC.




Allbery                                                        [Page 10]

                                 remctl                     October 2011


   9.   Server runs the command, collects the output, and sends the
        output back with flags TOKEN_DATA and the following payload
        format: four-octet exit status, four-octet data length, data.
        All numbers are in network byte order.  The exit status is 0 if
        there were no errors and non-zero otherwise, where the meaning
        of non-zero values are defined by the application.  The payload
        MUST be protected with gss_wrap with a conf_req_flag set to non-
        zero.

   10.  Server and client close connection.









































Allbery                                                        [Page 11]

                                 remctl                     October 2011


4.  Security Considerations

   It would be preferrable to insist on replay and sequence protection
   (GSS_C_REPLAY_FLAG and GSS_C_SEQUENCE_FLAG) for all contexts, but
   some older Kerberos GSS-API implementations don't support this and
   hence it is not mandatory in the protocol.  Clients SHOULD always
   request replay and sequence protection, however, and servers MAY
   require such protection be negotiated.

   The old protocol doesn't provide integrity protection for the flags,
   but since it always follows the same fixed sequence of operations,
   this should pose no security concerns in practice.  The new protocol
   only uses the flag field outside of the encrypted section of the
   packet for initial negotiation and closes the connection if the flags
   aren't what was expected (avoiding a down-negotiation attack).

   In the old protocol, the server calculated and sent a MIC back to the
   client, which then verified that the command as received by the
   server was correct.  Not only does GSS-API already provide integrity
   protection, but this verification also happens after the server has
   already started running the command.  It has been dropped in the new
   protocol.

   The old protocol doesn't require the client and server check the
   results of the GSS-API flag negotiation, although all old protocol
   clients passed GSS_C_MUTUAL_FLAG.  However, the old protocol requires
   gss_wrap be used for all payload with conf_req_flag set to non-zero,
   so any context that didn't negotiate confidentiality and integrity
   services would fail later.






















Allbery                                                        [Page 12]

                                 remctl                     October 2011


Appendix A.  Acknowledgements

   The original remctl protocol design was done by Anton Ushakov, with
   input from Russ Allbery and Roland Schemers.  Thank you to David
   Hoffman and Mike Newton for their review of the version 2 remctl
   protocol.













































Allbery                                                        [Page 13]

                                 remctl                     October 2011


Author's Address

   Russ Allbery
   Stanford University
   255 Panama Street, MC 4136
   Stanford, CA  94305-4136
   US

   Email: rra@stanford.edu
   URI:   http://www.eyrie.org/~eagle/









































Allbery                                                        [Page 14]

