1: <?xml version="1.0" standalone="no"?>
    2: <!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
    3: "http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd"
    4: [
    5: ]>
    6: 
    7: <article id="index">
    8:   <articleinfo>
    9:     <title>D-Bus Test Plan</title>
   10:     <date>14 February 2003</date>
   11:     <authorgroup>
   12:       <author>
   13:         <firstname>Anders</firstname>
   14:         <surname>Carlsson</surname>
   15:         <affiliation>
   16:           <orgname>CodeFactory AB</orgname>
   17:           <address><email>andersca@codefactory.se</email></address>
   18:         </affiliation>
   19:       </author>
   20:     </authorgroup>
   21:   </articleinfo>
   22:   <sect1 id="introduction">
   23:     <title>Introduction</title>
   24:     <para>
   25:       This document tries to explain the details of the test plan for D-Bus
   26:     </para>
   27:     <sect2 id="importance-of-testing">
   28:       <title>The importance of testing</title>
   29:       <para>
   30:         As with any big library or program, testing is important. It
   31:         can help find bugs and regressions and make the code better
   32:         overall. 
   33:       </para>
   34:       <para>
   35:         D-Bus is a large and complex piece of software (about 25,000
   36:         lines of code for the client library, and 2,500 lines of code
   37:         for the bus daemon) and it's therefore important to try to make sure
   38:         that all parts of the software is functioning correctly.
   39:       </para>
   40:       <para>
   41:         D-Bus can be built with support for testing by passing
   42:         <literal>--enable-tests</literal>. to the configure script. It
   43:         is recommended that production systems build without testing
   44:         since that reduces the D-Bus client library size.
   45:       </para>
   46:     </sect2>
   47:   </sect1>
   48:   <sect1 id="client-library">
   49:     <title>Testing the D-Bus client library</title>
   50:     <para>
   51:       The tests for the client library consist of the dbus-test
   52:       program which is a unit test for all aspects of the client
   53:       library. Whenever a bug in the client library is found and
   54:       fixed, a test is added to make sure that the bug won't occur again.
   55:     </para>
   56:     <sect2 id="data-structures">
   57:       <title>Data Structures</title>
   58:       <para>
   59:       The D-Bus client library consists of some data structures that
   60:       are used internally; a linked list class, a hashtable class and
   61:       a string class. All aspects of those are tested by dbus-test.
   62:       </para>
   63:     </sect2>
   64:     <sect2 id="message-loader">
   65:       <title>Message loader</title>
   66:       <para>
   67:         The message loader is the part of D-Bus that takes messages in
   68:         raw character form and parses them, turning them into DBusMessages.
   69:       </para>
   70:       <para>
   71:         This is one of the parts of D-Bus that
   72:         <emphasis>must</emphasis> be absolutely bug-free and
   73:         robust. The message loader should be able to handle invalid
   74:         and incomplete messages without crashing. Not doing so is a
   75:         serious issue and can easily result in D-Bus being exploitable
   76:         to DoS attacks.
   77:       </para>
   78:       <para>
   79:         To solve these problems, there is a testing feature called the
   80:         Message Builder. The message builder can take a serialized
   81:         message in string-form and convert it into a raw character
   82:         string which can then be loaded by the message loader. 
   83:       </para>
   84:       <figure>
   85:         <title>Example of a message in string form</title>
   86:         <programlisting>
   87:           # Standard org.freedesktop.DBus.Hello message
   88: 
   89:           VALID_HEADER
   90:           FIELD_NAME name
   91:           TYPE STRING
   92:           STRING 'org.freedesktop.DBus.Hello'
   93:           FIELD_NAME srvc
   94:           TYPE STRING
   95:           STRING 'org.freedesktop.DBus'
   96:           ALIGN 8
   97:           END_LENGTH Header
   98:           START_LENGTH Body
   99:           END_LENGTH Body
  100:         </programlisting>
  101:       </figure>
  102:       <para>
  103:         The file format of messages in string form is documented in
  104:         the D-Bus Reference Manual.
  105:       </para>
  106:       <para>
  107:         The message test part of dbus-test is using the message
  108:         builder to build different kinds of messages, both valid,
  109:         invalid, and invalid ones, to make sure that the loader won't
  110:         crash or leak memory of any of those, and that the loader
  111:         knows if a message is valid or not.
  112:       </para>
  113:       <para>
  114:         There is also a test program called
  115:         <literal>break-loader</literal> that loads a message in
  116:         string-form into raw character form using the message
  117:         builder. It then randomly changes the message, it can for
  118:         example replace single bytes of data or modify the length of
  119:         the message. This is to simulate network errors. The
  120:         break-loader program saves all the messages leading to errors
  121:         so it can easily be run for a long period of time.
  122:       </para>
  123:     </sect2>
  124:     <sect2 id="authentication">
  125:       <title>Authentication</title>
  126:       <para>
  127:         For testing authentication, there is a testing feature that
  128:         can read authentication sequences from a file and play them
  129:         back to a dummy server and client to make sure that
  130:         authentication is working according to the specification.
  131:       </para>
  132:       <figure>
  133:         <title>Example of an authentication script</title>
  134:         <programlisting>
  135:           ## this tests a successful auth of type EXTERNAL
  136:           
  137:           SERVER
  138:           SEND 'AUTH EXTERNAL USERNAME_HEX'
  139:           EXPECT_COMMAND OK
  140:           EXPECT_STATE WAITING_FOR_INPUT
  141:           SEND 'BEGIN'
  142:           EXPECT_STATE AUTHENTICATED
  143:         </programlisting>
  144:       </figure>
  145:     </sect2>
  146:   </sect1>
  147:   <sect1 id="daemon">
  148:     <title>Testing the D-Bus bus daemon</title>
  149:     <para>
  150:       Since the D-Bus bus daemon is using the D-Bus client library it
  151:       will benefit from all tests done on the client library, but
  152:       there is still the issue of testing client-server communication.
  153:       This is more complicated since it it may require another process
  154:       running.
  155:     </para>
  156:     <sect2 id="debug-transport">
  157:       <title>The debug transport</title>
  158:       <para>
  159:         In D-Bus, a <emphasis>transport</emphasis> is a class that
  160:         handles sending and receiving raw data over a certain
  161:         medium. The transport that is used most in D-Bus is the UNIX
  162:         transport with sends and recevies data over a UNIX socket. A
  163:         transport that tunnels data through X11 client messages is
  164:         also under development.
  165:       </para>
  166:       <para>
  167:         The D-Bus debug transport is a specialized transport that
  168:         works in-process. This means that a client and server that
  169:         exists in the same process can talk to eachother without using
  170:         a socket.
  171:       </para>
  172:     </sect2>
  173:     <sect2 id="bus-test">
  174:       <title>The bus-test program</title>
  175:       <para>
  176:         The bus-test program is a program that is used to test various
  177:         parts of the D-Bus bus daemon; robustness and that it conforms
  178:         to the specifications.
  179:       </para>
  180:       <para>
  181:         The test program has the necessary code from the bus daemon
  182:         linked in, and it uses the debug transport for
  183:         communication. This means that the bus daemon code can be
  184:         tested without the real bus actually running, which makes
  185:         testing easier.
  186:       </para>
  187:       <para>
  188:         The bus-test program should test all major features of the
  189:         bus, such as service registration, notification when things
  190:         occurs and message matching.
  191:       </para>
  192:     </sect2>
  193:   </sect1>
  194:   <sect1 id="other-tests">
  195:     <title>Other tests</title>
  196: 
  197:     <sect2 id="oom-robustness">
  198:       <title>Out-Of-Memory robustness</title>
  199:       <para>
  200:         Since D-Bus should be able to be used in embedded devices, and
  201:         also as a system service, it should be able to cope with
  202:         low-memory situations without exiting or crashing.
  203:       </para>
  204:       <para>
  205:         In practice, this means that both the client and server code
  206:         must be able to handle dbus_malloc returning NULL. 
  207:       </para>
  208:       <para>
  209:         To test this, two environment variables
  210:         exist. <literal>DBUS_MALLOC_FAIL_NTH</literal> will make every
  211:         nth call to dbus_malloc return NULL, and
  212:         <literal>DBUS_MALLOC_FAIL_GREATER_THAN</literal> will make any
  213:         dbus_malloc call with a request for more than the specified
  214:         number of bytes fail.
  215:       </para>
  216:     </sect2>
  217: 
  218:     <sect2 id="leaks-and-other-stuff">
  219:       <title>Memory leaks and code robustness</title> 
  220:       <para>
  221:         Naturally there are some things that tests can't be written
  222:         for, for example things like memory leaks and out-of-bounds
  223:         memory reading or writing.
  224:       </para>
  225:       <para>
  226:         Luckily there exists good tools for catching such errors. One
  227:         free good tool is <ulink url="http://devel-home.kde.org/~sewardj/">Valgrind</ulink>, which runs the program in a
  228:         virtual CPU which makes catching errors easy. All test programs can be run under Valgrind, 
  229:       </para>
  230:     </sect2>
  231:   </sect1>
  232: </article>